ERIC Number: EJ1439724
Record Type: Journal
Publication Date: 2024
Pages: 15
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0895-7347
EISSN: EISSN-1532-4818
New Tests of Rater Drift in Trend Scoring
John R. Donoghue; Carol Eckerly
Applied Measurement in Education, v37 n3 p225-239 2024
Trend scoring constructed response items (i.e. rescoring Time A responses at Time B) gives rise to two-way data that follow a product multinomial distribution rather than the multinomial distribution that is usually assumed. Recent work has shown that the difference in sampling model can have profound negative effects on statistics usually used to monitor rater drift. In this paper, three statistics, termed E-statistics, that account for the product-multinomial sampling by comparing conditional distributions, are introduced. A simulation compares performance with the paired t-test and Stuart's Q in detecting rater drift. Both the paired t-test and Q suffered extreme Type I error inflation for certain rescore study designs. The new E-statistics maintained good Type I error control and had good power to detect rater drift across occasions.
Descriptors: Scoring, Error of Measurement, Reliability, Scoring Rubrics, Responses, Statistical Analysis, Data Analysis, Interrater Reliability, Scores, Trend Analysis, Sampling, Design
Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Evaluative
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A