ERIC Number: EJ1355666
Record Type: Journal
Publication Date: 2022-Dec
Pages: 35
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0002-8312
EISSN: EISSN-1935-1011
Examining Human and Automated Ratings of Elementary Students' Writing Quality: A Multivariate Generalizability Theory Application
Chen, Dandan; Hebert, Michael; Wilson, Joshua
American Educational Research Journal, v59 n6 p1122-1156 Dec 2022
We used multivariate generalizability theory to examine the reliability of hand-scoring and automated essay scoring (AES) and to identify how these scoring methods could be used in conjunction to optimize writing assessment. Students (n = 113) included subsamples of struggling writers and non-struggling writers in Grades 3-5 drawn from a larger study. Students wrote six essays across three genres. All essays were hand-scored by four raters and an AES system called Project Essay Grade (PEG). Both scoring methods were highly reliable, but PEG was more reliable for non-struggling students, while hand-scoring was more reliable for struggling students. We provide recommendations regarding ways of optimizing writing assessment and blending hand-scoring with AES.
Descriptors: Reliability, Scoring, Essays, Automation, Writing Evaluation, Elementary School Students, Grade 3, Grade 4, Grade 5, Evaluation Methods, Natural Language Processing, Computer Assisted Testing, Writing Difficulties
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://bibliotheek.ehb.be:2993
Publication Type: Journal Articles; Reports - Research
Education Level: Elementary Education; Early Childhood Education; Grade 3; Primary Education; Grade 4; Intermediate Grades; Grade 5; Middle Schools
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A