NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
ERIC Number: ED464962
Record Type: Non-Journal
Publication Date: 2002-Apr
Pages: 25
Abstractor: N/A
ISBN: N/A
ISSN: N/A
EISSN: N/A
Score Dependability of the Writing and Speaking Sections of New TOEFL.
Lee, Yong-Won; Kantor, Robert; Mollaun, Pam
This study examines the score dependability of writing and speaking assessments from the Test of English as a Foreign Language (TOEFL) from the perspectives of univariate and multivariate generalizability theory (G-theory) and presents the findings of three separate G-theory studies. For writing, the focus was on evaluating the impact on reliability of the number of raters (or ratings) per essay (one or two) and the number of tasks (one, two, or three). For speaking, the focus was on investigating the impact of the number of tasks (1 to 12) and the number of ratings (1 or 2). In addition, an attempt was made to determine the optimal configuration of types and number of tasks for speaking to maximize the reliability of the composite score. Data for the first study were ratings from a study of a new task type in which 2 reading-writing and 3 listening-writing tasks were administered to 488 English-as-a-Second-Language learners. For study 2, each of the essays for the tasks for one subgroup was rerated by each of six raters. In the third study, focusing on speaking, data from 261 examinees were used. By far the greatest source of variation in examinees' test performance was due to differences among test takers' ability as measured by the writing and speaking tasks. This suggests that, as intended, the tasks do distinguish among examinees. Results further suggest that, to maximize score reliability for both speaking and writing, it would be more cost efficient to increase the number of tasks rather than the number of ratings per task. Beyond five or six tasks, there would be a diminishing return for increasing the number of tasks. (Contains 5 tables, 6 figures, and 12 references.) (SLD)
Publication Type: Numerical/Quantitative Data; Reports - Research; Speeches/Meeting Papers
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Educational Testing Service, Princeton, NJ.
Identifiers - Assessments and Surveys: Test of English as a Foreign Language
Grant or Contract Numbers: N/A