Abstract:
The potential of artificial intelligence (AI) in transforming education has received considerable attention. This study aims to explore the potential of large language mo...Show MoreMetadata
Abstract:
The potential of artificial intelligence (AI) in transforming education has received considerable attention. This study aims to explore the potential of large language models (LLMs) in assisting students with studying and passing standardized exams, while many people think it is a hype situation. Using primary education as an example, this research investigates whether ChatGPT-3.5 can achieve satisfactory performance on the Chinese Primary School Exams and whether it can be used as a teaching aid or tutor. We designed an experimental framework and constructed a benchmark that comprises 4800 questions collected from 48 tasks in Chinese elementary education settings. Through automatic and manual evaluations, we observed that ChatGPT-3.5’s pass rate was below the required level of accuracy for most tasks, and the correctness of ChatGPT-3.5’s answer interpretation was unsatisfactory. These results revealed a discrepancy between the findings and our initial expectations. However, the comparative experiments between ChatGPT-3.5 and ChatGPT-4 indicated significant improvements in model performance, demonstrating the potential of using LLMs as a teaching aid. This article also investigates the use of the trans-prompting strategy to reduce the impact of language bias and enhance question understanding. We present a comparison of the models' performance and the improvement under the trans-lingual problem decomposition prompting mechanism. Finally, we discuss the challenges associated with the appropriate application of AI-driven language models, along with future directions and limitations in the field of AI for education.
Published in: IEEE Transactions on Learning Technologies ( Volume: 17)
Funding Agency:
School of Computer Science, Shenyang Aerospace University, Shenyang, China
Yu Bai (Member, IEEE) received the master's degree in computer science and technology from Shenyang Aerospace University, Shenyang, China, in 2008.
He is an Associate Professor with Shenyang Aerospace University. He is also currently the CKO with Global Envoy Software Company Ltd., Shenyang. His research interests include artificial intelligence, natural language processing, information retrieval, and knowledge graph.
Prof....Show More
Yu Bai (Member, IEEE) received the master's degree in computer science and technology from Shenyang Aerospace University, Shenyang, China, in 2008.
He is an Associate Professor with Shenyang Aerospace University. He is also currently the CKO with Global Envoy Software Company Ltd., Shenyang. His research interests include artificial intelligence, natural language processing, information retrieval, and knowledge graph.
Prof....View more
School of Computer Science, Shenyang Aerospace University, Shenyang, China
Jun Li is currently working toward the graduate degree in computer science and technology with the School of Computer Science, Shenyang Aerospace University, Shenyang, China.
He is currently developing machine learning algorithms and human–machine cooperation solutions in prompt engineering and large language models (LLMs) optimizing. His research interests include LLMs and knowledge graph.
Jun Li is currently working toward the graduate degree in computer science and technology with the School of Computer Science, Shenyang Aerospace University, Shenyang, China.
He is currently developing machine learning algorithms and human–machine cooperation solutions in prompt engineering and large language models (LLMs) optimizing. His research interests include LLMs and knowledge graph.View more
School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, Australia
Jun Shen (Senior Member, IEEE) received the Ph.D. degree in computer application from Southeast University, Nanjing, China, in 2001.
Before 2006, he was at the Swinburne University of Technology, Melbourne, VIC, Australia, and the University of South Australia, Adelaide, SA, Australia. He is currently an Associate Professor with the School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, ...Show More
Jun Shen (Senior Member, IEEE) received the Ph.D. degree in computer application from Southeast University, Nanjing, China, in 2001.
Before 2006, he was at the Swinburne University of Technology, Melbourne, VIC, Australia, and the University of South Australia, Adelaide, SA, Australia. He is currently an Associate Professor with the School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, ...View more
School of Computer Science, Shenyang Aerospace University, Shenyang, China
Liang Zhao received the Ph.D. degree in computing from the School of Computing, Edinburgh Napier University, Edinburgh, U.K., in 2011.
He worked as an Associate Senior Researcher at Hitachi Research and Development Corporation, Hitachi, from 2012 to 2014. He is currently a Professor with Shenyang Aerospace University, Shenyang, China. He has authored and coauthored more than 150 articles. His research interests include int...Show More
Liang Zhao received the Ph.D. degree in computing from the School of Computing, Edinburgh Napier University, Edinburgh, U.K., in 2011.
He worked as an Associate Senior Researcher at Hitachi Research and Development Corporation, Hitachi, from 2012 to 2014. He is currently a Professor with Shenyang Aerospace University, Shenyang, China. He has authored and coauthored more than 150 articles. His research interests include int...View more
Contents School of Computer Science, Shenyang Aerospace University, Shenyang, China
Yu Bai (Member, IEEE) received the master's degree in computer science and technology from Shenyang Aerospace University, Shenyang, China, in 2008.
He is an Associate Professor with Shenyang Aerospace University. He is also currently the CKO with Global Envoy Software Company Ltd., Shenyang. His research interests include artificial intelligence, natural language processing, information retrieval, and knowledge graph.
Prof. Bai is currently a Member of the Language and Knowledge Computing Profession Committee of the Chinese Information Processing Society of China.
Yu Bai (Member, IEEE) received the master's degree in computer science and technology from Shenyang Aerospace University, Shenyang, China, in 2008.
He is an Associate Professor with Shenyang Aerospace University. He is also currently the CKO with Global Envoy Software Company Ltd., Shenyang. His research interests include artificial intelligence, natural language processing, information retrieval, and knowledge graph.
Prof. Bai is currently a Member of the Language and Knowledge Computing Profession Committee of the Chinese Information Processing Society of China.View more
School of Computer Science, Shenyang Aerospace University, Shenyang, China
Jun Li is currently working toward the graduate degree in computer science and technology with the School of Computer Science, Shenyang Aerospace University, Shenyang, China.
He is currently developing machine learning algorithms and human–machine cooperation solutions in prompt engineering and large language models (LLMs) optimizing. His research interests include LLMs and knowledge graph.
Jun Li is currently working toward the graduate degree in computer science and technology with the School of Computer Science, Shenyang Aerospace University, Shenyang, China.
He is currently developing machine learning algorithms and human–machine cooperation solutions in prompt engineering and large language models (LLMs) optimizing. His research interests include LLMs and knowledge graph.View more
School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, Australia
Jun Shen (Senior Member, IEEE) received the Ph.D. degree in computer application from Southeast University, Nanjing, China, in 2001.
Before 2006, he was at the Swinburne University of Technology, Melbourne, VIC, Australia, and the University of South Australia, Adelaide, SA, Australia. He is currently an Associate Professor with the School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, Australia, where he had been the Head of Postgraduate Studies and the Chair of School Research Committee since 2014. He has authored and coauthored more than 320 papers in journals and conferences in computer science/information technology areas. His publications appeared at IEEE Transactions on Learning Technologies and IEEE Transactions on Services Computing, Briefs in Bioinformatics, International Journal of Information Management, and many others. His research interests include computational intelligence, bioinformatics, cloud computing, and learning technologies including Massive Open Online Courses (MOOC).
Dr. Shen has been the Editor, Program Committee (PC) Chair, Guest Editor, and the PC Member for numerous journals and conferences published by IEEE, ACM, Elsevier, and Springer. He is a Senior Member of ACM and ACS. He was also a Member of ACM/AIS Task Force on Curriculum Masters of Science in Information Systems (MSIS) 2016.
Jun Shen (Senior Member, IEEE) received the Ph.D. degree in computer application from Southeast University, Nanjing, China, in 2001.
Before 2006, he was at the Swinburne University of Technology, Melbourne, VIC, Australia, and the University of South Australia, Adelaide, SA, Australia. He is currently an Associate Professor with the School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, Australia, where he had been the Head of Postgraduate Studies and the Chair of School Research Committee since 2014. He has authored and coauthored more than 320 papers in journals and conferences in computer science/information technology areas. His publications appeared at IEEE Transactions on Learning Technologies and IEEE Transactions on Services Computing, Briefs in Bioinformatics, International Journal of Information Management, and many others. His research interests include computational intelligence, bioinformatics, cloud computing, and learning technologies including Massive Open Online Courses (MOOC).
Dr. Shen has been the Editor, Program Committee (PC) Chair, Guest Editor, and the PC Member for numerous journals and conferences published by IEEE, ACM, Elsevier, and Springer. He is a Senior Member of ACM and ACS. He was also a Member of ACM/AIS Task Force on Curriculum Masters of Science in Information Systems (MSIS) 2016.View more
School of Computer Science, Shenyang Aerospace University, Shenyang, China
Liang Zhao received the Ph.D. degree in computing from the School of Computing, Edinburgh Napier University, Edinburgh, U.K., in 2011.
He worked as an Associate Senior Researcher at Hitachi Research and Development Corporation, Hitachi, from 2012 to 2014. He is currently a Professor with Shenyang Aerospace University, Shenyang, China. He has authored and coauthored more than 150 articles. His research interests include intelligent transportation system (ITS), vehicular Ad hoc network (VANET), wireless mesh network (WMN), and software defined network (SDN).
Dr. Zhao is a JSPS Invitational Fellow since 2023. He was listed as top 2\% of scientists in the world by Standford University (2022). He served as the Chair of several international conferences and workshops, including 2022 IEEE BigDataSE (Steering Cochair), 2021 IEEE TrustCom (Program Cochair), 2019 IEEE International Conferences on Ubiquitous Computing and Communications (IEEE IUCC) (Program Cochair), and 2018–2022 NGDN workshop (founder). He is an Associate Editor for Frontiers in Communications and Networking and the Journal of Circuits Systems and Computers. He is/has been a Guest Editor for IEEE Transactions on Network Science and Engineering, Springer Journal of Computing, etc. He was the recipient of the Best/Outstanding Paper Awards at 2015 IEEE IUCC, 2020 IEEE International Symposium on Parallel and Distributed Processing with Applications, 2022 IEEE International Conference on Embedded and Ubiquitous Computing, and 2013 ACM conference on mobile computing and multimedia (ACM MoMM).
Liang Zhao received the Ph.D. degree in computing from the School of Computing, Edinburgh Napier University, Edinburgh, U.K., in 2011.
He worked as an Associate Senior Researcher at Hitachi Research and Development Corporation, Hitachi, from 2012 to 2014. He is currently a Professor with Shenyang Aerospace University, Shenyang, China. He has authored and coauthored more than 150 articles. His research interests include intelligent transportation system (ITS), vehicular Ad hoc network (VANET), wireless mesh network (WMN), and software defined network (SDN).
Dr. Zhao is a JSPS Invitational Fellow since 2023. He was listed as top 2\% of scientists in the world by Standford University (2022). He served as the Chair of several international conferences and workshops, including 2022 IEEE BigDataSE (Steering Cochair), 2021 IEEE TrustCom (Program Cochair), 2019 IEEE International Conferences on Ubiquitous Computing and Communications (IEEE IUCC) (Program Cochair), and 2018–2022 NGDN workshop (founder). He is an Associate Editor for Frontiers in Communications and Networking and the Journal of Circuits Systems and Computers. He is/has been a Guest Editor for IEEE Transactions on Network Science and Engineering, Springer Journal of Computing, etc. He was the recipient of the Best/Outstanding Paper Awards at 2015 IEEE IUCC, 2020 IEEE International Symposium on Parallel and Distributed Processing with Applications, 2022 IEEE International Conference on Embedded and Ubiquitous Computing, and 2013 ACM conference on mobile computing and multimedia (ACM MoMM).View more
