Abstract:
Manually scoring and revising student essays has long been a time-consuming task for educators. With the rise of natural language processing techniques, automated essay s...Show MoreMetadata
Abstract:
Manually scoring and revising student essays has long been a time-consuming task for educators. With the rise of natural language processing techniques, automated essay scoring (AES) and automated essay revising (AER) have emerged to alleviate this burden. However, current AES and AER models require large amounts of training data and lack generalizability, which makes them hard to implement in daily teaching activities. Moreover, online sites offering AES and AER services charge high fees and have security issues uploading student content. In light of these challenges and recognizing the advancements in large language models (LLMs), we aim to fill these research gaps by analyzing the performance of open-source LLMs when accomplishing AES and AER tasks. Using a human-scored essay dataset (n = 600) collected in an online assessment, we implemented zero-shot, few-shot, and p-tuning AES methods based on the LLMs and conducted a human–machine consistency check. We conducted a similarity test and a score difference test for the results of AER with LLMs support. The human–machine consistency check result shows that the performance of open-source LLMs with a 10 B parameter size in the AES task is close to that of some deep-learning baseline models, and it can be improved by integrating the comment with the score into the shot or training continuous prompts. The similarity test and score difference test results show that open-source LLMs can effectively accomplish the AER task, improving the quality of the essays while ensuring that the revision results are similar to the original essays. This study reveals a practical path to cost-effectively, time-efficiently, and content-safely assisting teachers with student essay scoring and revising using open-source LLMs.
Published in: IEEE Transactions on Learning Technologies ( Volume: 17)
Funding Agency:

Research Center of Distance Education, Beijing Normal University, Beijing, China
Yishen Song received the B.Sc. degree in educational technology from Beijing Normal University, Beijing, China, in 2022, where he is currently working toward the M.Phil. degree in distance education with the Research Center of Distance Education.
His research interests include learning analytics and artificial intelligence in education.
Yishen Song received the B.Sc. degree in educational technology from Beijing Normal University, Beijing, China, in 2022, where he is currently working toward the M.Phil. degree in distance education with the Research Center of Distance Education.
His research interests include learning analytics and artificial intelligence in education.View more

National Engineering Research Center for Cyberlearning and Intelligent Technology, Beijing Normal University, Beijing, China
Qianta Zhu received the B.Eng. and M.Eng. degrees in software engineering from the Beijing University of Posts and Telecommunications, Beijing, China, in 2006 and 2018, respectively.
He is currently a Senior Research Officer with the National Engineering Research Center for Cyberlearning and Intelligent Technology, Beijing Normal University, Beijing. His research interests include artificial intelligence in education, Big ...Show More
Qianta Zhu received the B.Eng. and M.Eng. degrees in software engineering from the Beijing University of Posts and Telecommunications, Beijing, China, in 2006 and 2018, respectively.
He is currently a Senior Research Officer with the National Engineering Research Center for Cyberlearning and Intelligent Technology, Beijing Normal University, Beijing. His research interests include artificial intelligence in education, Big ...View more

School of Systems Science, Beijing Normal University, Beijing, China
Huaibo Wang received the B.Sc. and M.Sc. degrees in educational technology from Jiangsu Normal University, Xuzhou, China, in 2013 and 2016, respectively, and the Ph.D. degree in distance education from Beijing Normal University, Beijing, China, in 2021.
He is currently a Postdoctoral Fellow with the School of Systems Science, Beijing Normal University. His research interests include learning analytics, education complex sy...Show More
Huaibo Wang received the B.Sc. and M.Sc. degrees in educational technology from Jiangsu Normal University, Xuzhou, China, in 2013 and 2016, respectively, and the Ph.D. degree in distance education from Beijing Normal University, Beijing, China, in 2021.
He is currently a Postdoctoral Fellow with the School of Systems Science, Beijing Normal University. His research interests include learning analytics, education complex sy...View more

Research Center of Distance Education, Beijing Normal University, Beijing, China
Qinhua Zheng received the B.Sc. degree in educational technology, the M.Sc. degree in educational technology, and the Ph.D. degree in educational economy and management from Beijing Normal University, Beijing, China, in 2000, 2003, and 2006, respectively.
He is currently a Professor with Beijing Normal University. His research interests include online education, artificial intelligence in education, and Big Data in educati...Show More
Qinhua Zheng received the B.Sc. degree in educational technology, the M.Sc. degree in educational technology, and the Ph.D. degree in educational economy and management from Beijing Normal University, Beijing, China, in 2000, 2003, and 2006, respectively.
He is currently a Professor with Beijing Normal University. His research interests include online education, artificial intelligence in education, and Big Data in educati...View more

Research Center of Distance Education, Beijing Normal University, Beijing, China
Yishen Song received the B.Sc. degree in educational technology from Beijing Normal University, Beijing, China, in 2022, where he is currently working toward the M.Phil. degree in distance education with the Research Center of Distance Education.
His research interests include learning analytics and artificial intelligence in education.
Yishen Song received the B.Sc. degree in educational technology from Beijing Normal University, Beijing, China, in 2022, where he is currently working toward the M.Phil. degree in distance education with the Research Center of Distance Education.
His research interests include learning analytics and artificial intelligence in education.View more

National Engineering Research Center for Cyberlearning and Intelligent Technology, Beijing Normal University, Beijing, China
Qianta Zhu received the B.Eng. and M.Eng. degrees in software engineering from the Beijing University of Posts and Telecommunications, Beijing, China, in 2006 and 2018, respectively.
He is currently a Senior Research Officer with the National Engineering Research Center for Cyberlearning and Intelligent Technology, Beijing Normal University, Beijing. His research interests include artificial intelligence in education, Big Data in education, and multimodal learning analytics.
Qianta Zhu received the B.Eng. and M.Eng. degrees in software engineering from the Beijing University of Posts and Telecommunications, Beijing, China, in 2006 and 2018, respectively.
He is currently a Senior Research Officer with the National Engineering Research Center for Cyberlearning and Intelligent Technology, Beijing Normal University, Beijing. His research interests include artificial intelligence in education, Big Data in education, and multimodal learning analytics.View more

School of Systems Science, Beijing Normal University, Beijing, China
Huaibo Wang received the B.Sc. and M.Sc. degrees in educational technology from Jiangsu Normal University, Xuzhou, China, in 2013 and 2016, respectively, and the Ph.D. degree in distance education from Beijing Normal University, Beijing, China, in 2021.
He is currently a Postdoctoral Fellow with the School of Systems Science, Beijing Normal University. His research interests include learning analytics, education complex system, and student comprehensive quality evaluation.
Huaibo Wang received the B.Sc. and M.Sc. degrees in educational technology from Jiangsu Normal University, Xuzhou, China, in 2013 and 2016, respectively, and the Ph.D. degree in distance education from Beijing Normal University, Beijing, China, in 2021.
He is currently a Postdoctoral Fellow with the School of Systems Science, Beijing Normal University. His research interests include learning analytics, education complex system, and student comprehensive quality evaluation.View more

Research Center of Distance Education, Beijing Normal University, Beijing, China
Qinhua Zheng received the B.Sc. degree in educational technology, the M.Sc. degree in educational technology, and the Ph.D. degree in educational economy and management from Beijing Normal University, Beijing, China, in 2000, 2003, and 2006, respectively.
He is currently a Professor with Beijing Normal University. His research interests include online education, artificial intelligence in education, and Big Data in education.
Qinhua Zheng received the B.Sc. degree in educational technology, the M.Sc. degree in educational technology, and the Ph.D. degree in educational economy and management from Beijing Normal University, Beijing, China, in 2000, 2003, and 2006, respectively.
He is currently a Professor with Beijing Normal University. His research interests include online education, artificial intelligence in education, and Big Data in education.View more