ERIC Number: EJ1180138
Record Type: Journal
Publication Date: 2018-May
Pages: 8
Abstractor: As Provided
ISBN: N/A
ISSN: EISSN-1531-7714
EISSN: N/A
Rounding in Angoff Ratings
Wyse, Adam E.
Practical Assessment, Research & Evaluation, v23 n6 May 2018
One common modification to the Angoff standard-setting method is to have panelists round their ratings to the nearest 0.05 or 0.10 instead of 0.01. Several reasons have been offered as to why it may make sense to have panelists round their ratings to the nearest 0.05 or 0.10. In this article, we examine one reason that has been suggested, which is that even if panelists are given the opportunity to provide ratings to the nearest 0.01 they often round their ratings to the nearest 0.05 or 0.10 anyway. Using data from four standard settings, we show that in many cases ratings ended in a 0 or 5 when panelists were given the option of using a scale from 0 to 100 in one-point increments and that only about 9% of all ratings ended in a digit other than a 0 or 5. We also examined the impact of different rounding rules and we found that results were quite similar when using different rounding rules. Additional analyses showed the common phenomenon of panelists giving too high of ratings for hard items and too low of ratings for easy items in comparison to conditional p-values. It is suggested that rounding ratings to the nearest 0.05 or 0.10 represent reasonable alternatives to rounding ratings to the nearest 0.01.
Descriptors: Interrater Reliability, Evaluation Criteria, Scoring Formulas, Achievement Rating, Rating Scales, Cutting Scores, Standard Setting (Scoring), Medical Education, Correlation, Credentials
Center for Educational Assessment. 813 North Pleasant Street, Amherst, MA 01002. e-mail: pare@umass.edu; Tel: 413-577-2180; Web site: https://scholarworks.umass.edu/pare
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A