ERIC Number: EJ1413027
Record Type: Journal
Publication Date: 2023
Pages: 23
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1449-3098
EISSN: EISSN-1449-5554
Race with the Machines: Assessing the Capability of Generative AI in Solving Authentic Assessments
Binh Nguyen Thanh; Diem Thi Hong Vo; Minh Nguyen Nhat; Thi Thu Tra Pham; Hieu Thai Trung; Son Ha Xuan
Australasian Journal of Educational Technology, v39 n5 p59-81 2023
In this study, we introduce a framework designed to help educators assess the effectiveness of popular generative artificial intelligence (AI) tools in solving authentic assessments. We employed Bloom's taxonomy as a guiding principle to create authentic assessments that evaluate the capabilities of generative AI tools. We applied this framework to assess the abilities of ChatGPT-4, ChatGPT-3.5, Google Bard and Microsoft Bing in solving authentic assessments in economics. We found that generative AI tools perform very well at the lower levels of Bloom's taxonomy while still maintaining a decent level of performance at the higher levels, with "create" being the weakest level of performance. Interestingly, these tools are better able to address numeric-based questions than text-based ones. Moreover, all the generative AI tools exhibit weaknesses in building arguments based on theoretical frameworks, maintaining the coherence of different arguments and providing appropriate references. Our study provides educators with a framework to assess the capabilities of generative AI tools, enabling them to make more informed decisions regarding assessments and learning activities. Our findings demand a strategic reimagining of educational goals and assessments, emphasising higher cognitive skills and calling for a concerted effort to enhance the capabilities of educators in preparing students for a rapidly transforming professional environment.
Descriptors: Artificial Intelligence, Models, Performance Based Assessment, Economics Education, Introductory Courses, Natural Language Processing, Assignments, Evaluation, Educational Objectives, Behavioral Objectives, Classification, Integrity, Higher Education
Australasian Society for Computers in Learning in Tertiary Education. Ascilite Secretariat, P.O. Box 44, Figtree, NSW, Australia. Tel: +61-8-9367-1133; e-mail: info@ascilite.org.au; Web site: https://ajet.org.au/index.php/AJET
Publication Type: Journal Articles; Reports - Research; Tests/Questionnaires
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A