ERIC Number: EJ1238530
Record Type: Journal
Publication Date: 2019-Dec
Pages: 27
Abstractor: As Provided
ISBN: N/A
ISSN: EISSN-2330-8516
EISSN: N/A
Prediction of Writing True Scores in Automated Scoring of Essays by Best Linear Predictors and Penalized Best Linear Predictors. Research Report. ETS RR-19-13
Yao, Lili; Haberman, Shelby J.; Zhang, Mo
ETS Research Report Series, Dec 2019
Many assessments of writing proficiency that aid in making high-stakes decisions consist of several essay tasks evaluated by a combination of human holistic scores and computer-generated scores for essay features such as the rate of grammatical errors per word. Under typical conditions, a summary writing score is provided by a linear combination of the holistic scores and the feature scores. The best linear predictor (BLP) is used to approximate the true composite writing score by a linear combination of holistic scores and scores of essay features. However, because the relationship between computer-generated feature score and human scores may depend on subgroup membership and the same scoring rules must normally be applied to all test takers, Yao, Haberman, and Zhang proposed a modified methodology of the penalized best linear predictor (PBLP) by incorporating a quadratic penalty function into the conventional BLP method. This research report contains full accounts of the BLP results as well as supplementary PBLP results to Yao et al. for three assessments of writing that aid in making high-stakes decisions: the "TOEFL iBT"® Writing test, the "GRE"® General Analytical Writing subject test, and the "Praxis"® Core Academic Skills for Educators: Writing assessment. Results obtained indicate the added value in using machine features for prediction of composite true scores of essay writings and effectiveness of the penalty function in suppressing the lack of population invariance.
Descriptors: Prediction, True Scores, Computer Assisted Testing, Scoring, Essays, Writing Evaluation, High Stakes Tests, English (Second Language), Language Tests, Second Language Learning, Graduate Study, College Entrance Examinations, Writing Tests, Accuracy
Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Publication Type: Journal Articles; Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Test of English as a Foreign Language; Graduate Record Examinations; Praxis Series
Grant or Contract Numbers: N/A