ERIC Number: EJ1111378
Record Type: Journal
Publication Date: 2008-Jan
Pages: 84
Abstractor: As Provided
ISBN: N/A
ISSN: EISSN-2330-8516
EISSN: N/A
Analytic Scoring of TOEFL® CBT Essays: Scores from Humans and "E-rater"®. TOEFL® Research Reports. RR-81. ETS RR-08-01
Lee, Yong-Won; Gentile, Claudia; Kantor, Robert
ETS Research Report Series, Jan 2008
The main purpose of the study was to investigate the distinctness and reliability of analytic (or multitrait) rating dimensions and their relationships to holistic scores and "e-rater"® essay feature variables in the context of the TOEFL® computer-based test (CBT) writing assessment. Data analyzed in the study were analytic and holistic essay scores provided by human raters and essay feature variable scores computed by "e-rater" (version 2.0) for two TOEFL CBT writing prompts. It was found that (a) all of the six analytic scores were not only correlated among themselves but also correlated with the holistic scores, (b) high correlations obtained among holistic and analytic scores were largely attributable to the impact of essay length on both analytic and holistic scoring, (c) there may be some potential for profile scoring based on analytic scores, and (d) some strong associations were confirmed between several "e-rater" variables and analytic ratings. Implications are discussed for improving the analytic scoring of essays, validating automated scores, and refining "e-rater" essay feature variables.
Descriptors: English (Second Language), Language Tests, Second Language Learning, Scoring, Essays, Writing Tests, Computer Assisted Testing, Computer Software, Cues, Prompting, Correlation, Holistic Approach, Test Reliability, Evaluators, Profiles, Multitrait Multimethod Techniques, Statistical Analysis
Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: Test of English as a Foreign Language
Grant or Contract Numbers: N/A