ERIC Number: EJ1432383
Record Type: Journal
Publication Date: 2024-Jul
Pages: 16
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1475-7257
EISSN: EISSN-1475-7257
Comparing ChatGPT with Experts' Responses to Scenarios That Assess Psychological Literacy
M. Anthony Machin; Tanya M. Machin; Natalie Gasson
Psychology Learning and Teaching, v23 n2 p265-280 2024
Progress in understanding students' development of psychological literacy is critical. However, generative AI represents an emerging threat to higher education which may dramatically impact on student learning and how this learning transfers to their practice. This research investigated whether ChatGPT responded in ways that demonstrated psychological literacy and whether it matched the responses of subject matter experts (SMEs) on a measure of psychological literacy. We tasked ChatGPT with providing responses to 13 psychology research methods scenarios as well as to rate each of the five response options that were already developed for each scenario by the research team. ChatGPT responded in ways that would typically be regarded as displaying a high level of psychological literacy. The response options which were previously rated by two groups of SMEs were then compared with ratings provided by ChatGPT. The Pearson's correlations were very high (r's = 0.73 and 0.80, respectively), as were the Spearman's rhos (rho's = 0.81 and 0.82, respectively). Kendall's tau were also quite high (tau's = 0.67 and 0.68, respectively). We conclude that ChatGPT may generate responses that match SME psychological literacy in research methods, which could also generalise across multiple domains of psychological literacy.
Descriptors: Psychology, Higher Education, Artificial Intelligence, Intelligent Tutoring Systems, Research Methodology, Knowledge Level
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://bibliotheek.ehb.be:2993
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A