ERIC Number: EJ1429749
Record Type: Journal
Publication Date: 2024
Pages: 18
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: EISSN-1833-2595
Can You Spot the Bot? Identifying AI-Generated Writing in College Essays
Tal Waltzer; Celeste Pilegard; Gail D. Heyman
International Journal for Educational Integrity, v20 Article 11 2024
The release of ChatGPT in 2022 has generated extensive speculation about how Artificial Intelligence (AI) will impact the capacity of institutions for higher learning to achieve their central missions of promoting learning and certifying knowledge. Our main questions were whether people could identify AI-generated text and whether factors such as expertise or confidence would predict this ability. The present research provides empirical data to inform these speculations through an assessment given to a convenience sample of 140 college instructors and 145 college students (Study 1) as well as to ChatGPT itself (Study 2). The assessment was administered in an online survey and included an AI Identification Test which presented pairs of essays: In each case, one was written by a college student during an in-class exam and the other was generated by ChatGPT. Analyses with binomial tests and linear modeling suggested that the AI Identification Test was challenging: On average, instructors were able to guess which one was written by ChatGPT only 70% of the time (compared to 60% for students and 63% for ChatGPT). Neither experience with ChatGPT nor content expertise improved performance. Even people who were confident in their abilities struggled with the test. ChatGPT responses reflected much more confidence than human participants despite performing just as poorly. ChatGPT responses on an AI Attitude Assessment measure were similar to those reported by instructors and students except that ChatGPT rated several AI uses more favorably and indicated substantially more optimism about the positive educational benefits of AI. The findings highlight challenges for scholars and practitioners to consider as they navigate the integration of AI in education.
Descriptors: Artificial Intelligence, Man Machine Systems, Natural Language Processing, College Students, College Faculty, Identification, Prediction, Expertise, Self Esteem, Writing (Composition), Essays
BioMed Central, Ltd. Available from: Springer Nature. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: customerservice@springernature.com; Web site: https://bibliotheek.ehb.be:2129/gp/biomedical-sciences
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: National Science Foundation (NSF)
Authoring Institution: N/A
Grant or Contract Numbers: 2104610
Data File: URL: https://osf.io/2c54a/