ERIC Number: ED585968
Record Type: Non-Journal
Publication Date: 2014
Pages: 34
Abstractor: As Provided
ISBN: N/A
ISSN: EISSN-
EISSN: N/A
Linguistic Microfeatures to Predict L2 Writing Proficiency: A Case Study in Automated Writing Evaluation
Crossley, Scott A.; Kyle, Kristopher; Allen, Laura K.; Guo, Liang; McNamara, Danielle S.
Grantee Submission, The Journal of Writing Assessment v7 n1 2014
This study investigates the potential for linguistic microfeatures related to length, complexity, cohesion, relevance, topic, and rhetorical style to predict L2 writing proficiency. Computational indices were calculated by two automated text analysis tools (Coh- Metrix and the Writing Assessment Tool) and used to predict human essay ratings in a corpus of 480 independent essays written for the TOEFL. A stepwise regression analysis indicated that six linguistic microfeatures explained 60% of the variance in human scores for essays in a test set, providing an exact accuracy of 55% and an adjacent accuracy of 96%. To examine the limitations of the model, a post-hoc analysis was conducted to investigate differences in the scoring outcomes produced by the model and the human raters for essays with score differences of two or greater (N = 20). Essays scored as high by the regression model and low by human raters contained more word types and perfect tense forms compared to essays scored high by humans and low by the regression model. Essays scored high by humans but low by the regression model had greater coherence, syntactic variety, syntactic accuracy, word choices, idiomaticity, vocabulary range, and spelling accuracy as compared to essays scored high by the model but low by humans. Overall, findings from this study provide important information about how linguistic microfeatures can predict L2 essay quality for TOEFL-type exams and about the strengths and weaknesses of automatic essay scoring models.
Descriptors: Computational Linguistics, Essays, Scoring, Writing Evaluation, Second Language Learning, Prediction, Writing Skills, English (Second Language), Language Tests, Evaluators, Accuracy, Comparative Analysis, Syntax, Connected Discourse, Language Usage, Vocabulary Skills, Figurative Language, Cues
Publication Type: Journal Articles; Reports - Research; Tests/Questionnaires
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Test of English as a Foreign Language
IES Funded: Yes
Grant or Contract Numbers: R305A080589; R305G020018