Which one? AI-assisted language assessment or paper format: an exploration of the impacts on foreign language anxiety, learning attitudes, motivation, and writing performance

Abstract

In recent years, language practitioners have paid increasing attention to artificial intelligence (AI)’s role in language programs. This study investigated the impact of AI-assisted language assessment on L2 learners’ foreign language anxiety (FLA), attitudes, motivation, and writing skills. The study adopted a sequential exploratory mixed-methods design. Divided between an experimental group (receiving AI-assisted assessment) and a control group (receiving paper-format assessment), the participants were 70 intermediate English learners from two intact university classes in Bangladesh. The TOEFL iBT writing section measured writing skills, while the study also investigated perceptions and experiences of FLA, attitudes, and motivation using narrative frames. Thematic analysis of the narrative data showed that AI-assisted assessment greatly raised learners’ motivation, improved attitudes about language acquisition, and lowered FLA. According to quantitative analysis, the pretest writing abilities across groups showed no appreciable variation. Even though the difference was not statistically significant on the posttest, the experimental group exceeded the control group. The results of this study imply that AI-assisted assessments can generate a helpful learning environment, lower anxiety, improve attitudes, and increase motivation, thereby delivering useful information. Future studies should investigate long-term consequences, and further improvements to AI tools should optimize educational advantages—attitudes, motivation, and writing skills.

Introduction

In recent years, the use of information technology in language education and assessment has attracted significant interest from researchers (Ahmadi, 2018; Lei et al., 2022; Liu, 2023; Namaziandost & Rezai, 2024; Shadiev & Yang, 2020; Shadiev et al., 2023; Soleimani et al., 2022). Incorporating information technology into language teaching can greatly enhance the learning experience by providing personalized, interactive, and communicative learning opportunities (Chun et al., 2016; Rahimi & Fathi, 2022; Rodinadze & Zarbazoia, 2012; Shatri, 2020). Educators use information technology to develop virtual learning environments that engage students and support language acquisition (Fathi & Rahimi, 2022; Loncar et al., 2023; Nguyen & Le, 2023). Notably, AI has become a valuable tool in language education, improving student learning outcomes (Haristiani, 2019; Huang et al., 2023; Knox, 2020; Pedro et al., 2019; Pikhart, 2020).

FLA, a critical emotional factor in learning foreign languages, has been a major focus of research since the 1970s. In psychological terms, anxiety is described as an unpleasant mental state characterized by feelings of nervousness, fear, and worry, which is triggered by the autonomic nervous system (Spielberger, 1972). FLA specifically pertains to anxiety experienced when learning a foreign language (Horwitz et al., 1986; Jiang et al., 2022; MacIntyre, 1995; Sımsek & Capar, 2024). Horwitz et al. (1986) defined FLA as a unique combination of self-perceptions, beliefs, emotions, and behaviors associated with the distinctive challenges of classroom language learning.

In a broad sense, attitude refers to an individual’s internal states that influence their potential actions (Cai et al., 2023; Droba, 1933). Within an academic context, this concept relates to how students view their teachers, peers, and learning environment (Masgoret & Gardner, 2003). Based on this definition, Thang et al. (2011) described English students’ attitudes as their positive or negative perceptions of their English instructors, courses, and the learning process. According to Feng and Hong (2022), language learners with positive attitudes are more likely to engage enthusiastically in language acquisition. Consequently, students with positive attitudes tend to be more successful in learning a new language (Prastiwi, 2018). Conversely, learners with negative attitudes invest less time and effort into language acquisition (Moghadam & Shamsi, 2021) and often struggle to master the target language (Guo et al., 2023). Research has shown a strong correlation between language learners’ attitudes toward the learning process and classroom environment with their academic motivation (Genc & Aydin, 2017; Rasool & Winke, 2019), language achievement (Dewaele & Proietti Ergün, 2020; Prastiwi, 2018), and success in acquiring a second language (Hashemian & Heidari, 2013; Paker & Erarslan, 2015).

Language learning motivation, as described by Dörnyei (2009), involves both personal factors (such as previous learning experiences and perceptions of the target language’s utility) and social factors (such as attitudes toward the target language’s societal status) that shape one’s attitudes and behaviors toward learning it. In a different view, Deci and Ryan (1995) emphasize the importance of intrinsic and extrinsic motivation for learning outcomes. Learners driven by intrinsic motivation typically find the learning process enjoyable and experience high satisfaction and low anxiety. These positive learning experiences often lead to increased intrinsic motivation. Conversely, some learners depend on external factors, such as rewards and punishments, to motivate their engagement in learning (see Teng & Wu, 2024).

Writing skills are essential in both personal and professional spheres to meet the demands of the twenty-first century. For instance, writing is the basis for developing key skills like reading, listening, and speaking (Abubakr Abdulrahman & Kara, 2022; Artayasa et al., 2018). Furthermore, writing accurate responses to exam questions is crucial for evaluating learners’ abilities through both summative and formative assessments (Rahman et al., 2021). With their chat features, social media platforms also require users to communicate their ideas effectively and clearly. Additionally, consistent writing practice can boost learners’ imagination, creativity, and problem-solving abilities, as each writing exercise involves a constructive cycle of ideation, composition, and decision-making (Fajriah, 2023; Kara & Abdulrahman, 2022).

Although information technology is becoming increasingly integrated into language education and assessment, knowledge of how AI-assisted language assessment affects important and performance-related aspects of language acquisition is known. Crucially affecting students’ achievement and whole experience are FLA, learning attitudes, motivation, and writing performance. Long the norm for assessing language proficiency, it is unclear whether traditional paper-based tests are as effective as AI-assisted tests, especially in reducing FLA and improving motivation and writing skills. The shift from paper-based to AI-powered tests offers both chances and difficulties. In addition to lowering grading biases, AI-assisted assessments can generate more engaging learning environments and provide personalized feedback. However, there is limited data on whether these possible advantages are lower anxiety, better attitudes and motivation, and greater writing performance among language learners.

Furthermore, how such technology changes students’ psychological and emotional states is unexplored. By investigating the relative impacts of paper-based language examinations and AI-assisted ones on FLA, learning attitudes, motivation, and writing performance, this study aims to close these gaps. Teachers and policymakers trying to improve language learning results and develop conducive, efficient learning contexts must first understand these effects. The results could shed important light on how to maximize AI-assisted assessment to help language learners and whether they may present a reasonable substitute for traditional paper-format approaches.

This study is significant as it addresses a critical gap in understanding the effects of AI-assisted language assessment on learners’ emotional and performance-related outcomes, such as FLA, attitudes, motivation, and writing skills. As AI becomes increasingly integrated into educational settings, exploring whether these tools can offer advantages over traditional paper-based assessments is crucial. The findings have potential implications for various stakeholders in language pedagogy. For teachers, AI-assisted assessments can create more engaging and supportive learning environments by reducing anxiety and enhancing student motivation. For learners, the personalized feedback and interactive nature of AI tools can improve writing skills and overall language proficiency. For syllabus designers and policymakers, the results suggest that integrating AI into assessment practices can modernize language education, providing a balanced approach that better meets the needs of diverse learners. This study thus provides a foundation for future research and practical applications in optimizing language teaching and assessment through technology.

Literature review

Artificial intelligence in language programs

AI is a system comprising intelligent programs collaborating with humans to execute various tasks (Aldosari, 2020). In educational environments, AI can make decisions similar to human cognition (Akerkar, 2014). Researchers in applied linguistics have identified the promise of AI in the realms of language learning and teaching, aiming to improve instructional methods for educators and facilitate language acquisition for students (Luckin et al., 2016; Nazari et al., 2021; Sun et al., 2021; Xia et al., 2022; Zhang & Zou, 2023). AI-driven online platforms can generate essential language input and output, supporting learners in their language development. These tools, accessible via computers and mobile devices, are particularly beneficial for enhancing writing skills. A prominent AI tool is ChatGPT, an AI-assisted chatbot developed by OpenAI (Barrot, 2023). ChatGPT is effectively employed in various language learning courses to improve writing skills (Barrot, 2023). It provides extensive knowledge and generates grammatically correct sentences, producing coherent and cohesive text. This tool understands human queries and delivers suitable responses. Additionally, ChatGPT helps learners tackle writing issues related to organization, coherence, grammar, and vocabulary, offering alternative suggestions to correct ungrammatical sentences and enhance overall writing proficiency (see Song & Song, 2023).

Numerous studies have explored the impact of AI-enhanced language learning tools on the academic performance of English language learners (Divekar et al., 2022; Fitria, 2023; Suryana et al., 2020). For example, Zheng et al. (2023) examined AI’s influence on learning outcomes and perceptions in a meta-analysis. Analyzing 24 studies with 2908 participants from 2001 to 2020, they found that AI had a more substantial effect on learning achievement than on learners’ perceptions. This suggests that most research has focused on AI’s positive impact on academic performance, with learner perceptions being a secondary consideration. Additionally, Xu et al. (2023) investigated the effects of AI-assisted language learning on English learners’ speaking and interaction skills. Their study revealed that using AI tools with speech recognition features enhanced learners’ language achievement and actively engaged them in interactive language learning activities.

Ebadi and Amini (2024) explored the influence of AI-supported language learning on the engagement levels of EFL learners. They collected data through questionnaires measuring motivation, social presence, and human-likeness, alongside recording learner interactions with the AI tool. The study found that the AI tool significantly boosted learners’ motivation and engagement. Similarly, Carpio Cañada et al. (2015) assessed the impact of an AI-driven language learning approach on students’ motivation and academic performance, revealing that it enhanced learners’ motivation, improving their academic achievements. Additionally, Ali et al. (2023) investigated the effect of ChatGPT on the motivation of English language learners and teachers, finding that the tool notably improved writing and reading skills but had a neutral impact on speaking and listening abilities. Furthermore, Schmidt-Fajlik (2023) compared ChatGPT with Grammarly and ProWritingAid regarding their effectiveness in checking, understanding, and enhancing EFL learners’ English grammar. The results showed that ChatGPT outperformed the other AI tools in identifying and improving grammatical errors.

Yan (2023) explored the impact of an AI-assisted language learning tool on the writing abilities of EFL students. The study concluded that the AI tool significantly improved the students’ writing performance, enabling them to complete writing tasks more efficiently. Yan highlighted that the tool’s effectiveness in accelerating task completion underscored its value in writing activities. Similarly, Utami et al. (2023) investigated the influence of AI-powered language learning on three Indonesian EFL learners’ academic research writing skills through a case study approach. Data collected via questionnaires and interviews indicated that the AI tool provided valuable feedback, comments, and alternative sentences, enhancing students’ academic writing and engagement. Lee et al. (2023) examined the effects of an AI-assisted language learning tool on EFL learners’ reading enjoyment. One group of learners used the AI tool to generate reading topics based on their interests, while another group participated in traditional reading comprehension activities. The results demonstrated that the AI tool significantly increased the learners’ reading enjoyment.

Foreign language anxiety

FLA has been a significant area of research in foreign language education since the 1970s. Anxiety, from a psychological standpoint, is described as an uncomfortable mental state marked by nervousness, fear, and worry triggered by the autonomic nervous system (Spielberger, 1972). FLA is a specific type of anxiety that arises in the context of learning a foreign language (Chen et al., 2024; Horwitz et al., 1986; MacIntyre, 1995). According to Horwitz et al. (1986), FLA encompasses a distinct blend of self-perceptions, beliefs, emotions, and behaviors related to the unique difficulties encountered in classroom language learning.

Horwitz et al. (1986) were pioneers in examining FLA as a distinct phenomenon. To address the limitations of previous research methodologies related to FLA, they developed the Foreign Language Classroom Anxiety Scale (FLCAS). This innovation marked the end of an era where FLA studies lacked standardized measurement tools (Guo & Xu, 2014), signaling a maturation in FLA research. Consequently, scholars began exploring FLA’s overall impact and its correlations with various factors (Aida, 1994; Young, 1986, 1992), as well as its connection to fundamental language skills like listening, speaking, reading, and writing (Hu et al., 2024; Sellers, 2000; Vogely, 1998). In essence, FLA involves tension, fear, and nervousness within self-awareness, emotions, beliefs, and behaviors (Aida, 1994; Pan & Zhang, 2023), triggered by situations requiring an unfamiliar foreign language (MacIntyre & Gardner, 1991).

Attitudes

In general, attitude encompasses an individual’s internal dispositions that shape their potential behaviors (Droba, 1933). This concept pertains to how students perceive their educators, classmates, and the overall learning environment (Masgoret & Gardner, 2003). Thang et al. (2011) characterized English students’ attitudes as favorable or unfavorable views of their English teachers, courses, and the learning process. Feng and Hong (2022) noted that language learners with positive attitudes are more inclined to participate actively in language acquisition. Consequently, these learners are generally more successful in mastering a new language (Prastiwi, 2018). On the other hand, students with negative attitudes tend to dedicate less time and effort to language learning (Moghadam & Shamsi, 2021) and often face difficulties in achieving proficiency in the target language (Guo et al., 2023). Studies have demonstrated a significant link between learners’ attitudes toward the learning process and classroom environment with their academic motivation (Genc & Aydin, 2017; Rasool & Winke, 2019), language achievement (Dewaele & Proietti Ergün, 2020; Prastiwi, 2018), and overall success in second language acquisition (Hashemian & Heidari, 2013; Paker & Erarslan, 2015).

John Oller and his team undertook several extensive studies examining the link between attitudes and language proficiency (Chihara & Oller, 1978; Oller et al., 1977, 1978). Their research focused on how Chinese, Japanese, and Mexican students’ success in learning English correlated with their attitudes toward themselves, their native language groups, the target language group, their motivations for learning English, and their reasons for coming to the USA. Generally, positive attitudes toward oneself, the native language group, and the target language group were found to improve proficiency. However, the studies yielded mixed results regarding the benefits and drawbacks of integrative versus instrumental motivations. For instance, one study revealed that students who did not intend to stay in the USA permanently achieved higher proficiency levels.

Motivation

Dörnyei (2009) defines language learning motivation as encompassing both personal elements (like past learning experiences and beliefs about the usefulness of the target language) and social elements (such as views on the societal standing of the target language) that influence one’s attitudes and behaviors toward language acquisition. On another note, Deci and Ryan (1995) highlight the critical role of intrinsic and extrinsic motivation in learning outcomes. Learners motivated intrinsically usually find joy in the learning process, resulting in high satisfaction and low anxiety, which further fuels their intrinsic motivation. In contrast, some learners rely on external stimuli, such as rewards or penalties, to drive their learning efforts (see Teng & Wu, 2024).

Dörnyei (2009) introduced the second language (L2) motivational self-system to provide a clearer understanding of motivation. This framework includes three components: the Ideal Self, the Ought-to Self, and the L2 Learning Experience. The Ideal Self drives L2 learners to engage in learning activities to close the gap between their current abilities and their envisioned language proficiency. The Ought-to Self encompasses the qualities that individuals feel they should have. The L2 Learning Experience comprises the chosen learning methods, the teacher’s instructional approach, and peer support (for a fuller review, see Namaziandost et al., 2024). Research generally indicates that the Ideal Self positively influences the L2 learning process, whereas the Ought-to Self tends to have a less favorable or sometimes negative effect on students’ motivation to learn a language (Man et al., 2018; Ushioda & Dörnyei, 2009; Yashima, 2009; You & Dörnyei, 2016).

Regarding L2 motivation in online learning environments, researchers like Cai and Zhu (2012) have noted that virtual learning experiences enhance learners’ L2 Learning Experience but do not significantly impact the Ideal or Ought-to Self. This suggests that the L2 Learning Experience is more adaptable and can be quickly influenced. Another study by Lee and Lu (2023) found that, compared to the Ought-to Self, the Ideal Self is a stronger predictor of learners’ willingness to communicate in online settings. A positive Ideal Self can benefit online learning by improving learners’ psychological and emotional states, reducing anxiety, and increasing interest. This, in turn, may encourage learners to be more engaged and proactive in online education.

The three components of the Motivational Self System are fundamentally linked to an individual’s self-efficacy beliefs (Man et al., 2018). Research has shown that perceived self-efficacy fosters motivation and serves as a significant predictor of learner performance in educational activities (Bandura & Schunk, 1981; Glynn et al., 2011; Shea & Bidjerano, 2010), including those conducted online (Kim et al., 2014; Teng & Yue, 2023). A majority of studies have focused on the connection between traditional learning and assessment environments, yet the connection between AI-assisted assessment and language learning motivation is not well-grounded.

Writing skills

Writing skills are vital for success in both personal and professional contexts in the twenty-first century. For example, writing is foundational for developing other essential skills such as reading, listening, and speaking (Abubakr Abdulrahman & Kara, 2022; Artayasa et al., 2018). Additionally, providing accurate written responses in exams is key for assessing students’ capabilities through summative and formative evaluations. With their chat features, social media platforms necessitate the clear and effective expression of ideas. Moreover, regular writing practice enhances learners’ imagination, creativity, and problem-solving skills, as each writing task involves a productive cycle of generating ideas, composing text, and making decisions (Fajriah, 2023; Kara & Abdulrahman, 2022).

Academic writing courses have garnered significant attention in educational institutions, aiming to provide learners with the necessary skills to succeed in academic writing gradually and ultimately. Three key methodologies—product, process, and genre approaches—have emerged as central to these courses, creating a professional and supportive learning environment. The product approach thoroughly analyzes model writing tasks, culminating in creating a new written work (Riberio, 2011; Yucedal et al., 2022). This method, also known as guided writing, enables learners to become progressively independent by encouraging brainstorming, discussion, elaboration, and refinement of ideas. Conversely, the process approach emphasizes collaborative brainstorming, drafting, and iterative editing based on feedback, making the writing process a continuous cycle of idea exchange and improvement (Menggo & Gunas, 2022; Strobel & Van Barneveld, 2009). This approach shifts the focus from a teacher-centered to a student-centered model, fostering a more participatory and interactive learning experience. Similarly, attention has recently been turned toward how AI-assisted instruction and assessment might influence L2 learners’ writing skills.

This study tackles little knowledge of how AI-assisted language assessment affects FLA, learning attitudes, motivation, and writing performance compared to conventional paper-based evaluations. Although AI tools such as ChatGPT have shown promise in improving language learning results, current studies mostly illustrate their benefits on academic achievement without closely investigating their impact on learners’ emotional and psychological states. This disparity is crucial as attitudes and motivation are fundamental to language acquisition FLA. Although the potential of AI-assisted exams to reduce anxiety, improve attitudes, and increase motivation and writing skills remains underexplored, traditional paper-based assessment may not sufficiently address these affective aspects. By looking at the relative advantages of these two assessment systems, this study hopes to close this gap and offer information to guide teachers and policymakers in besting language learning conditions. Thus, there following research questions are addressed in this study:

• 1. How do L2 learners perceive FLA as the result of AI-assisted language assessment?

• 2. What is L2 learners’ attitude toward AI-assisted language assessment?

• 3. How does L2 learners’ language learning motivation change due to AI-assisted language assessment?

• 4. How does AI-assisted language assessment affect L2 learners’ writing skills?

Method

Design

This study adopts a sequential exploratory mixed-methods design, starting with qualitative data to check L2 learners’ perception of FLA as the result of AI-assisted language assessment and their attitude toward AI-assisted language assessment. Additionally, the study explores how AI-assisted language assessment might change L2 learners’ motivation. In the quantitative phase, using quantitative data, the study attempts to uncover the potential of AI-assisted language assessment in enhancing L2 learners’ writing skills.

Participants

The participants in this study were selected from two intact classes at a university in Bangladesh, each comprising 35 learners, resulting in a total of 70 participants. An intact class refers to a pre-existing group of students already enrolled in the same course (Ary et al., 2019), ensuring that the sample represented a natural classroom without disrupting the students’ regular learning environment. This method was chosen to maintain the ecological validity of the study. Both classes were equivalent in language proficiency, as determined by the Oxford Quick Placement Test (OQPT), making them suitable for random assignment into the experimental and control groups. This approach ensured that the sample represented intermediate-level English language learners in that context. The age range of the participants was from 18 to 25 years. All participants were native Bengali speakers and were intermediate-level English language learners, as determined by the OQPT. The participants were randomly assigned to either experimental or control groups. The experimental group, consisting of 35 learners, received AI-assisted language assessments, while the control group, also consisting of 35 learners, received traditional paper-format assessments. This random assignment ensured that each group had an equivalent distribution of language proficiency levels, allowing for a fair comparison of the effects of the two different assessment formats on FLA, learning attitudes, motivation, and writing performance.

Instruments

Narrative frames were utilized to investigate the impact of AI-assisted language and paper-format assessments on FLA, learning attitudes, and language learning motivation. Narrative frames are structured templates that guide participants to reflect on and articulate their experiences and feelings related to language learning (Ary et al., 2019). These frames helped capture qualitative data on the participants’ emotional and attitudinal responses to the different assessment formats.

The writing section of the TOEFL iBT (Test of English as a Foreign Language Internet-based Test) was employed to evaluate the effect of the assessment types on writing skills. The TOEFL iBT writing section is a standardized test widely recognized for assessing English language writing proficiency (Cushing Weigle, 2002). It consists of two tasks: an integrated writing task, where test-takers read a passage, listen to a related lecture, and then write a response combining the information, and an independent writing task, where they write an essay based on a given topic (Cushing Weigle, 2002). The TOEFL iBT writing section provided a reliable and valid measure of the participants’ writing performance, allowing for a comprehensive evaluation of the differences between the AI-assisted and paper-format assessment groups.

These instruments were selected to ensure a robust analysis of the two assessment methods’ affective and performance-related outcomes, thereby addressing the study’s objectives comprehensively.

Data collection procedures

The data collection procedure for this study was meticulously designed to comprehensively assess the impact of AI-assisted language assessment and paper-format assessment on FLA, learning attitudes, motivation, and writing performance. Initially, all participants were tested using the OQPT to confirm their intermediate level of English proficiency. Subsequently, they were randomly assigned to either the experimental group (35 learners) or the control group (35 learners).

Before the treatment began, both groups completed narrative frames to document their initial levels of FLA, learning attitudes, and motivation, providing a baseline for post-treatment comparisons. Additionally, all participants completed the TOEFL iBT writing section to establish their baseline writing proficiency. The treatment phase lasted 8 weeks, during which the experimental group received AI-assisted language assessments and the control group received traditional paper-format assessments.

Participants in the experimental group used an AI-driven language assessment tool, specifically ChatGPT, integrated into their regular language learning activities. They received weekly writing assignments, which they submitted through the AI platform. The AI provided immediate feedback on various aspects of their writing, including organization, coherence, grammar, and vocabulary, and offered suggestions for improvement and alternative sentence structures. To ensure consistent use and provide technical support, participants engaged with the AI tool for at least 1 h per week in a monitored lab setting. Throughout the treatment, they were encouraged to reflect on their experiences using narrative frames, focusing on their anxiety levels, attitudes toward learning, and motivation.

In contrast, participants in the control group completed their writing assignments on paper, which were collected weekly and graded by their instructor. Written feedback from the instructor covered the same aspects as the AI tool: organization, coherence, grammar, and vocabulary. Weekly feedback sessions, lasting 1 h each, were held to discuss assignments and provide additional guidance, mirroring the time commitment of the experimental group. Participants in the control group also used narrative frames to document their reflections on anxiety, learning attitudes, and motivation during the treatment phase.

To further detail, the AI-assisted and paper-based assessments were carefully designed to ensure consistency in evaluating the participants’ language abilities while offering different delivery modes.

The AI-assisted assessments were administered through an AI-powered platform tailored to language learning. The platform provided automated, real-time feedback on various aspects of language use, including grammar, vocabulary, coherence, and organization in writing tasks. The assessments were interactive and adaptive, with questions adjusting in difficulty based on the learner’s responses. The AI system also offered personalized feedback and suggestions for improvement, making the assessment process more engaging and supportive for learners.

Conversely, the paper-based assessments followed a traditional format, where participants answered questions on printed sheets without any technological aid. The tests were designed to be equivalent in content and difficulty to the AI-assisted assessments. They included a series of writing prompts and comprehension questions to evaluate the same language skills. Responses were manually assessed by a language instructor, providing feedback similar in scope but lacking the immediacy and personalization of the AI system.

The AI-assisted assessments were designed to provide real-time feedback using an automated grading system that evaluated grammar, vocabulary, coherence, and writing quality. Learners received immediate corrective suggestions and alternative phrasing options through the AI tool, enabling them to revise their responses iteratively. In contrast, the paper-based assessments followed a traditional format where learners submitted written responses, which the instructor manually graded. This format lacked the immediate feedback feature of the AI-assisted system, relying instead on post-assessment evaluations provided days later.

Both assessment types aimed to measure participants’ writing performance but differed in delivery and feedback mechanisms. This design allowed for a comparative analysis of the impacts of AI-assisted versus traditional assessments on learners’ anxiety, attitudes, motivation, and writing performance.

After the 8-week treatment period, both groups completed the narrative frames again to capture any changes in FLA, learning attitudes, and motivation. Participants also retook the TOEFL iBT writing section to assess changes in their writing performance. The qualitative data from the narrative frames were analyzed to identify changes within and between the groups, while the quantitative data from the TOEFL iBT writing assessments were statistically analyzed to determine significant differences in writing performance between the experimental and control groups.

By following this detailed data collection procedure, the study aimed to thoroughly compare the effects of AI-assisted language assessment versus traditional paper-format assessment on key affective and performance-related factors in language learning. This approach ensured a robust analysis, capturing both the emotional and cognitive dimensions of the participants’ learning experiences.

Data analysis procedures

This study’s data analysis procedures involved qualitative and quantitative methods to comprehensively examine the effects of AI-assisted language assessment and paper-format assessment on FLA, learning attitudes, motivation, and writing performance.

A thematic analysis was conducted to analyze the impact of different types of assessment on FLA, learning attitudes, and motivation. First, the recorded narratives from the participants were transcribed manually. Once transcription was complete, the data was meticulously reviewed to derive themes that emerged from the participants’ reflections manually. This thematic analysis allowed for an in-depth understanding of the emotional and attitudinal changes experienced by the learners throughout the treatment period.

Independent-sample t-tests were performed to measure the effect of the different modes of assessment on writing skills. These statistical tests were used to compare the writing performance of the experimental and control groups on both the pretest and posttest. According to Ary et al. (2019), this inferential statistical test is used when the means of two groups are compared and contrasted. If the data are normally distributed, one is allowed to conduct the test. If not, one must resort to the non-parametric variant of the test (i.e., the Mann–Whitney U test). By conducting independent-samples t-tests, the study aimed to determine whether there were any significant differences in the writing scores between the groups before and after the treatment.

This dual approach of thematic analysis for qualitative data and independent-samples t-tests for quantitative data provided a robust framework for evaluating the multifaceted impacts of the assessment types on the participants. Combining these methods ensured a thorough analysis of different dimensions of language learning within the study.

Results

L2 learners’ perception of FLA as the result of AI-assisted language assessment

Over the 8 weeks, the outcomes of the narratives on L2 learners’ perception of FLA in the experimental group—who underwent AI-assisted language assessments—showered a clear drop in anxiety levels. Many participants admitted to being anxious and reluctant about writing assignments. Still, as they interacted with the AI tool, their narratives started to show comfort and assurance. The AI’s instant feedback and helpful advice helped ease their worries about blunders. The nonjudging character of the AI appealed to participants since it lessened the pressure usually connected with conventional tests. This encouraging setting lets students concentrate on content and creativity instead of anxiety of mistakes, promoting a more laid-back attitude toward writing. Most of the experimental group reported a notable drop in FLA at the end of the treatment, which they attributed to the tailored and consistent feedback from the AI tool, thereby lessening the anxiety and making the learning process more interesting.

By comparison, the control group—which underwent traditional paper-form assessments—showcased a distinct narrative about FLA. Although some participants first said they were accustomed to and comfortable with the conventional assessment approach, others indicated great degrees of anxiety and dread of judgment. For several of the subjects, the anxiety levels stayed rather constant throughout the trial. The narratives frequently emphasized the anxiety related to waiting for the teacher’s feedback, which was less instantaneous and occasionally seen as critical or demoralizing. This delay in getting feedback added to ongoing concerns about performance and expertise. Though the instructor’s feedback was comprehensive and targeted toward development, the personal nature of critiques often made students feel inadequate. The control group did not show the same degree of anxiety lowering by the end of the session as the experimental group. According to the narratives, several participants kept or improved their FLA using the traditional assessment technique since the pressure to perform well and the fear of negative assessment remained major issues.

Three main themes emerged from the narratives regarding L2 learners’ perceptions of FLA in the context of AI-assisted language assessment and paper-format assessment: Feedback Timeliness and Nature, Perceived Judgment and Pressure, and Learner Autonomy and Confidence.

Feedback Timeliness and Nature, as the first theme, underlined the importance of quick and helpful feedback in lowering FLA. Those in the experimental group often cited the advantages of getting quick feedback from the AI tool. The constant, nonjudging, encouraging tone of the feedback helped them repair mistakes immediately and gradually raise their writing abilities. On the other hand, the control group frequently voiced annoyance with the slow response from their teacher, which occasionally hampered their capacity to correct errors promptly and added to sustained anxiety.

Perceived Judgment and Pressure, the second theme, emphasized how their assessment might influence students’ anxiety levels. Using the AI tool gave the experimental group a safe environment for learning and was free from negative personal evaluation, reducing their perceived pressure and judgment. This non-human contact lessened the psychological load connected with traditional tests. Conversely, the narratives of the control group showed that the conventional paper-style tests sometimes increased their concern about unfavorable evaluation from the teacher, which resulted in sustained anxiety and tension connected to their performance.

Additionally, reducing FLA was mostly dependent on learner autonomy and confidence. The experimental group developed more confidence and autonomy by utilizing their interactions with the AI tool. The tailored recommendations and autonomy in editing their work helped them regain control over their learning process, therefore lowering their anxiety. On the other hand, the control group claimed to be less independent and more dependent on the feedback coming from their instructor, which occasionally affected their confidence and maintained their anxiety level.

L2 learners’ attitude toward AI-assisted language assessment

The narratives from the experimental group showcased quite favorable opinions of this kind of assessment. Emphasizing how the AI tool made the assessment process more interesting and less stressful, participants often underlined its simplicity and ease of use. The students thought the AI’s capacity to deliver quick, thorough feedback—which they considered to be both motivating and instructive—was valuable. Their quick feedback loop helped them notice and fix errors immediately, fostering a more dynamic and engaging learning environment. Since participants could work independently and review the results as needed, many said the AI-assisted assessment encouraged independence and self-directed learning. Furthermore, a major benefit of AI’s objectivity in grading was eliminating any possible bias and guaranteeing a fair assessment of their output. Citing more confidence, motivation, and a more positive attitude toward writing tasks, the experimental group indicated generally a high degree of satisfaction with the AI-assisted assessments.

By contrast, the control group showered conflicting opinions about the traditional approach. Appreciating the structured approach and direct instructor involvement, some participants indicated comfort and familiarity with paper-based exams. Still, many of the narratives revealed annoyance with the slow response that comes with traditional tests. Students often felt that the lag hindered their ability to correct errors and quickly raise their writing quality in feedback. Furthermore, some participants felt their attitude toward writing assignments suffered because of the instructor’s too critical or demoralizing comments. The pressure to do well in a one-time submission free of iterative feedback cycles caused anxiety and lowered motivation for several students. Although the control group appreciated tailored feedback from an experienced teacher, several voiced a wish for a more interactive and instantaneous assessment system similar to what AI tools offer. Generally, the views of the control group about traditional paper-style tests were less favorable than those of the experimental group toward assessments supported by AI.

Three main themes emerged from the narratives regarding L2 learners’ attitudes toward AI-assisted language and paper-format assessment: Feedback Quality and Timing, Perceived Fairness and Impartiality, and Engagement and Motivation.

The first theme, Feedback Quality and Timing, underlined how important quick and thorough feedback is for influencing students’ attitudes. Those in the experimental group stressed the advantages of getting swift, comprehensive feedback from AI technology. They appreciated the chance to spot and fix mistakes quickly, supporting sustained learning and development. On the other hand, the control group regularly complained about their instructor’s delayed responses, pointing out that it hampered their capacity for quick changes and enhancements.

Emphasizing the need for objective assessment in shaping students’ views, Perceived Fairness and Impartiality as the second theme, the experimental group valued the AI’s objectivity since it offered consistent and objective evaluations. One major benefit of this objectivity was guaranteed fair grading free from any possible human influence. On the other hand, some control group members said that depending on the instructor’s point of view, traditional tests could occasionally be subjective, which sometimes resulted in perceptions of injustice.

Engagement and Motivation, the third theme, concentrated on the degree of involvement of the students in the assessment process and their natural motivation. The interactive and user-friendly character of the AI tool reportedly raised the experimental group’s motivation and engagement. The instantaneous feedback and the flexibility to work at their speed helped them to develop autonomy and active participation in their learning process. Conversely, the control group showed different degrees of involvement and motivation; some members felt demotivated by the strain and anxiety connected with traditional paper-form tests. The lack of iterative feedback and the one-time submission style often produced a less engaging and inspiring experience.

L2 learners’ language learning motivation in response to AI-assisted language assessment

Language learning motivation showed a clear rise in the accounts from the experimental group. Participants often said that the instantaneous and tailored feedback of the AI tool greatly increased their motivation. With each iteration of feedback, the students could clearly show development in their writing abilities, which inspired them to participate in the learning process. Their natural drive was much strengthened by this cycle of constant improvement since it made them feel successful and forward. Their want to learn was further stoked by the dynamic and interesting character of the AI tool, which made learning more fun. The ability of the AI tool to let participants take control of their learning pace and style also helped many of them value their autonomy. This self-directed learning setting encouraged a greater dedication to raising their English competency and a more intense curiosity in language acquisition. The experimental group showed generally increased desire and passion for language learning, mostly attributing this positive shift to the responsive and supporting character of the AI-assisted assessment.

On the other hand, the narratives of the control group showed a more conflicting reaction to language learning motivation. Although some students valued the consistent and familiar format of paper tests, many said traditional assessments’ delayed feedback and one-time submission character negatively influenced their motivation. Lack of quick, thorough feedback sometimes left students unsure about their development and less driven to interact closely with the course of study. Some participants believed that their general motivation was lowered, and their anxiety was exacerbated by the pressure to do well on one assessment devoid of iterative feedback cycles. Further reducing their motivation was the apparent subjectivity of the instructor’s feedback, which occasionally caused resentment.

Notwithstanding these difficulties, some participants mentioned that direct contact with their teacher gave them useful, tailored feedback—which could inspire them. However, the disadvantages of delayed and less frequent input sometimes exceeded this advantage, leading to a generally lower degree of motivation than in the experimental group. Traditional paper-form tests had less overall influence on the motivation of the control group for language learning, which emphasizes the need for more interesting and responsive assessment strategies.

Three main themes emerged from the narratives regarding L2 learners’ language learning motivation in response to AI-assisted language and paper-format assessments: Immediate and Personalized Feedback, Autonomy and Engagement, and Anxiety and Perceived Fairness.

The first theme—Immediate and Personalized Feedback—emphasized how much timely, customized feedback affects students’ motivation. The participants in the experimental group underlined the motivating power of the instantaneous and thorough feedback of the AI technology. Their fast awareness of their errors and ability to make required corrections with this immediacy helped create a cycle of sustained learning and development. On the other hand, the control group frequently underlined the demotivating effect of delayed feedback in conventional paper tests, which left them doubtful about their growth and less likely to interact closely with the content.

The second theme, autonomy, and engagement, underlined how interactive learning environments and learner autonomy help to increase motivation. The experimental group valued the AI tool’s autonomy so they could learn at their speed and style. Their intrinsic motivation and enthusiasm for language learning were enhanced by this self-directed learning strategy, together with the interesting character of the AI tool. By contrast, the control group showed different degrees of participation; some students felt demotivated by the strict and less engaging character of paper-form tests.

The third theme, Anxiety, and Perceived Fairness, focused on the emotional and psychological aspects of the assessment process. The experimental group reported lower anxiety levels due to the impartial and consistent nature of AI assessments, which they perceived as fair and unbiased. This anxiety reduction contributed to a more positive and motivating learning experience. On the other hand, the control group often mentioned that the pressure of performing well in a single assessment and the potential subjectivity in instructor feedback increased their anxiety and negatively affected their motivation. The perceived lack of fairness in traditional assessments further dampened their enthusiasm for language learning.

The effect of AI-assisted language assessment on L2 learners’ writing skills

A t-test was conducted to measure the effect of different assessment modes on writing skills. Before performing the t-test, a Kolmogorov–Smirnov test verified the normality of the data.

As Table 1 demonstrates, the data was normally distributed on both the pretest and the posttest (p > 0.05).

Table 1 One-sample Kolmogorov–Smirnov test

Table 2 shows a similar performance between the experimental group (N = 35, M = 4.800, SD = 1.827) and the control group (N = 35, M = 4.428, SD = 1.719) on the pretest.

Table 2 Group statistics on the pretest

Table 3 shows a non-significant difference between the two assessment modes on the pretest (p > 0.05).

Table 3 Independent samples test on the pretest

Table 4 showcases that the experimental group (N = 35, M = 11.685, SD = 6.402) outperformed the control group (N = 35, M = 9.514, SD = 4.368) on the posttest.

Table 4 Group statistics on the posttest

Although the experimental group outfared their peers in the control condition on the posttest, the difference between the two groups was not significant (t = 1.658, df = 60.018, p > 0.05). The Cohen’s d effect size was 0.396, indicating a small effect size (Table 5).

Table 5 Independent samples test on the posttest

Discussion

The results of this study provide valuable insights into the impacts of AI-assisted language assessment compared to traditional paper-format assessment on various aspects of L2 learning, including writing skills, FLA, learning attitudes, and language learning motivation. This discussion synthesizes and relates these findings to existing literature, offering interpretations and implications for future research and educational practice.

The quantitative analyses of the writing abilities showed that the experimental group performed better than the control group on the posttest. Although the difference was not statistically significant, the better mean scores point to a possible advantage of AI assessments in improving writing skills. This is consistent with earlier studies stressing the benefits of AI in offering quick, customized feedback and encouraging constant development (Barrot, 2023; Song & Song, 2023). The AI-assisted assessment gave students thorough awareness of their writing mistakes and helpful advice for correction, which would help explain their superior results.

The qualitative analysis of the narratives highlighted complex views of FLA, learning attitudes, and motivation in response to AI-supported assessments. The experimental group for FLA reported reduced anxiety, which they attributed to the constant, objective character of AI feedback. This is in line with results from Ebadi and Amini (2024), who observed among students utilizing AI tools, less anxiety and higher motivation. Conversely, the control group felt more anxious because of the strain of conventional tests and the apparent subjectivity in grading, stressing a crucial area where AI might offer a more encouraging learning environment.

Regarding learning attitudes, the experimental group praised the dynamic and interesting character of the AI-assisted assessment. This contrasts with the varied opinions of the control group, which were shaped by the rigidity and lack of engagement in paper-format tests. These results confirm the hypothesis that, as advised by Feng and Hong (2022) and Genc and Aydin (2017), AI technologies can help to create an interesting and autonomous learning environment.

Regarding language learning motivation, the experimental group said the AI tool’s instantaneous and tailored feedback increased their motivation. Echoing the motivating results reported by Lee and Lu (2023) and Dörnyei (2009), this feedback loop promoted a more proactive and self-directed learning approach. However, the control group showed different degrees of motivation, usually reduced by delayed feedback and the less dynamic character of traditional tests. These findings underscore the need for timely and pertinent feedback in preserving and increasing student motivation.

Regarding writing ability, the experimental group performed better than the control group on the posttest. Although the difference was not statistically significant, the better mean scores point to a possible advantage of AI tools in improving writing ability. This is consistent with earlier studies showing that by offering broad knowledge and producing grammatically perfect phrases, AI-driven tools like ChatGPT can greatly improve writing abilities (Barrot, 2023; Song & Song, 2023). Furthermore, emphasized in research by Utami et al. (2023) and Yan (2023) is that immediate and customized feedback from AI tools is essential for the continual development of writing skills.

In a nutshell, the findings of this current study could well be aligned with the existing literature. For instance, previous research has highlighted the potential of AI tools like ChatGPT in enhancing learners’ writing skills and academic performance (Barrot, 2023; Yan, 2023), yet these studies have primarily focused on cognitive outcomes rather than affective factors such as FLA, attitudes, and motivation. The current study addresses this gap by demonstrating that AI-assisted assessments can reduce FLA and foster a more positive learning environment. This supports the findings by Ebadi and Amini (2024), who observed increased learner engagement and motivation with AI tools. Additionally, the study aligns with the work of Ali et al. (2023), who noted improvements in language skills, suggesting that AI can serve as a comprehensive tool that addresses cognitive and emotional needs in language learning. However, the study diverges from some literature, such as Xu et al. (2023), where the primary focus was on speaking and interaction skills.

In total, the results of this study generally highlight the possible advantages of AI-assisted language assessment in improving writing skills, lowering FLA, encouraging good learning attitudes, and increasing language learning motivation. These findings complement the more general body of research stressing the significance of AI in enhancing language learning results (Aldosari, 2020; Divekar et al., 2022; Fitria, 2023; Suryana et al., 2020; Zheng et al., 2023).

This study has profound implications for various L2 stakeholders. Including AI-assisted language assessment tools in their lessons will help language teachers as much as it will help others. According to the current study, AI systems can give students timely and customized feedback—vital for improving their writing abilities and general language proficiency. Teachers should think about using these tools to augment more traditional assessment systems. Doing this can lower the stress related to language tests since AI systems provide a consistent and objective assessment, enabling students to feel more confident and less concerned about their results. AI tools can also successfully include students, promoting a more participatory and learner-centered classroom. This integration can help free teachers’ time so they may concentrate more on customized instruction and support.

Relatedly, AI-assisted tools let language learners improve their learning experience. According to the study, these technologies lower FLA, increase motivation, and strengthen writing abilities by offering quick and constructive feedback. Students should be urged to practice and advance their language outside the classroom using AI tools as a supplemental resource—instant feedback and improvement suggestions help students participate in a more proactive and self-directed learning process. Moreover, the lower anxiety and higher motivation connected with AI assessments can result in a more pleasant and interesting educational environment, improving language learning results.

Designers of syllabi should keep the incorporation of AI-assisted assessments top importance in mind. The research results show how well AI tools could improve language learning utilizing tailored feedback and creating an interactive learning environment. Regular language assessment and practice components should be AI tools incorporated into courses developed by syllabus designers. This can entail creating projects and activities using AI capacity to offer real-time help and feedback. Furthermore, syllabus designers should ensure that the application of AI techniques fits the general goals and results of the language development. This will help them design a more efficient and all-encompassing language education program that satisfies the demands of various students and benefits from educational technology.

Conclusion

This study investigated how L2 learners’ FLA, attitudes, motivation, and writing abilities changed with AI-assisted language assessment. The results show that including AI tools in language assessment systems has major advantages. The lower FLA among students utilizing AI-assisted assessments emphasizes the possibility of these tools to produce a less demanding and more encouraging learning environment. AI’s instantaneous, tailored feedback seems to assist students in better controlling their anxiety than more conventional paper-form tests. AI-assisted assessment also positively impacted students’ perceptions of language learning. Increased engagement, perceived justice, and better self-efficacy, among other themes, point to how AI tools might change students’ impressions of their language learning process. Success in language acquisition and continuous involvement depend on this encouraging change in perspective. The study also indicated that AI-supported exams greatly increase students’ motivation. AI technologies’ interactive character and tailored feedback help to sustain high degrees of intrinsic motivation by motivating students to create objectives, self-regulate, and see the worth of their work. Success in long-term language acquisition depends on these inspiring advantages. Although the quantitative study of writing abilities revealed no statistically significant variation between the experimental and control groups, the better posttest performance of the AI-assisted group suggests possible long-term advantages. These results imply that AI-assisted tests might cause more significant improvements in writing skills with constant use.

The potential long-term impacts of AI-assisted language assessment on language learning outcomes might be enormous. AI-assisted assessments have the potential to create more personalized and adaptive learning environments, enabling learners to receive immediate, targeted feedback that can accelerate their language proficiency over time. These assessments could also reduce FLA and enhance motivation, resulting in more sustained and positive engagement with the language learning process. Notwithstanding, to fully capture these impacts, future research should focus on longitudinal studies that track learners’ progress and attitudes over extended periods. Additionally, researchers should assess the scalability of AI-assisted assessments across diverse linguistic and cultural contexts and their effects on different language skills, such as speaking and listening, which have received less attention in current studies. Investigating the integration of AI with traditional assessment methods to create a hybrid model might also offer significant insights into maximizing the benefits of both approaches.

This study shows the encouraging role AI may play in language assessment. Including AI techniques in language instruction can help lower learner anxiety, give more efficient and tailored feedback, and inspire good attitudes and motivation. AI-assisted tests generate interesting and motivating surroundings from which language learners could profit. Syllabus designers should consider AI-assisted tests to produce more active and successful language education programs.

The findings of this study are contextually bound to the specific sample and setting, which may affect their generalizability. As a result, the findings may not fully apply to learners from different linguistic or cultural backgrounds or those in varied educational environments. Future research could explore how these tools perform across various age groups, cultural contexts, and proficiency levels to provide a more comprehensive understanding of their broader impact. Future studies should also examine the long-term consequences of AI-assisted tests on different language skills and how these tools could be improved to maximize their learning value. Adopting AI technology in language programs can transform teaching and learning approaches, producing more efficient and enjoyable language learning environments and experiences.