ERIC Number: EJ1419734
Record Type: Journal
Publication Date: 2024
Pages: 32
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: EISSN-1946-6226
Available Date: N/A
Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment
ACM Transactions on Computing Education, v24 n1 Article 5 2024
Cheating has been a long-standing issue in university assessments. However, the release of ChatGPT and other free-to-use generative AI tools has provided a new and distinct method for cheating. Students can run many assessment questions through the tool and generate a superficially compelling answer, which may or may not be accurate. We ran a dual-anonymous "quality assurance" marking exercise across four end-of-module assessments across a distance university computer science (CS) curriculum. Each marker received five ChatGPT-generated scripts alongside 10 student scripts. A total of 90 scripts were marked; every ChatGPT generated script for the undergraduate modules received at least a passing grade (>40%), with all of the introductory module CS1 scripts receiving a distinction (>85%). None of the ChatGPT-taught postgraduate scripts received a passing grade (>50%). We also present the results of interviewing the markers and of running our sample scripts through a GPT-2 detector and the TurnItIn AI detector, which both identified every ChatGPT-generated script but differed in the number of false positives. As such, we contribute a baseline understanding of how the public release of generative AI is likely to significantly impact quality assurance processes. Our analysis demonstrates that in most cases, across a range of question formats, topics, and study levels, ChatGPT is at least capable of producing adequate answers for undergraduate assessment.
Descriptors: Computer Science Education, Artificial Intelligence, Cheating, Student Evaluation, Identification, Natural Language Processing, Integrity, Evaluation Methods, Computer Software, Automation, Quality Assurance, Student Behavior, Open Universities, Distance Education, Foreign Countries, Undergraduate Study, Graduate Study, Plagiarism
Association for Computing Machinery. 2 Penn Plaza Suite 701, New York, NY 10121. Tel: 800-342-6626; Tel: 212-626-0500; Fax: 212-944-1318; e-mail: acmhelp@acm.org; Web site: http://toce.acm.org/
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: United Kingdom
Grant or Contract Numbers: N/A
Author Affiliations: N/A