ERIC Number: EJ1309958
Record Type: Journal
Publication Date: 2021
Pages: 12
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0731-1745
EISSN: N/A
A Model-Data-Fit-Informed Approach to Score Resolution in Performance Assessments
Wind, Stefanie A.; Walker, A. Adrienne
Educational Measurement: Issues and Practice, v40 n3 p52-63 Fall 2021
Many large-scale performance assessments include score resolution procedures for resolving discrepancies in rater judgments. The goal of score resolution is conceptually similar to person fit analyses: To identify students for whom observed scores may not accurately reflect their achievement. Previously, researchers have observed that rater-agreement methods and person fit analyses result in similar conclusions about which students' achievement estimates warrant additional investigation, and that score resolution generally improves person fit. We consider the implications of using person fit analysis as an initial step to identify performances for score resolution, and of using fit indices to identify raters to provide additional ratings. We simulated student responses to multiple-choice items and a writing task. We simulated various types of person misfit in the writing task and identified the persons who needed resolution using a person-fit index. Results indicate larger improvements in person fit after resolution when the person fit approach was used compared to a rater-agreement approach. With the fit-informed approach, person fit improved for of the misfitting students; with the rater-agreement approach, person fit improved for around one third of these students. We consider the implications of our findings for mixed-format assessments.
Descriptors: Goodness of Fit, Performance Based Assessment, Evaluators, Decision Making, Scores, Interrater Reliability, Multiple Choice Tests, Comparative Analysis, Evaluation Methods, Simulation, Writing Evaluation
Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://bibliotheek.ehb.be:2191/en-us
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A