While much research has explored the quality of traditional teacher evaluations, little is known about the quality of ratings in preservice teacher evaluations. This paper presents a case study of Many-facet Rasch measurement (MFR, Linacre, 1989) to explore potential rater effects influencing the quality of supervisor ratings in educator preparation programs (EPPs) at a comprehensive four-year university in the Southern United States. Because it offers a fine-grained look at the reasonableness of ratings related to each aspect of the evaluation: the individual raters, the candidates, and the teaching subdomains (Wind & Jones, 2019), the authors apply MFR to demonstrate its usefulness in answering the following questions related to preservice teachers: (1) How self-consistent are preservice teacher supervisors when giving ratings? (2) How well do preservice teacher supervisors differentiate between candidates at differing levels of teaching proficiency? (3) How well do preservice teacher supervisors differentiate between teaching subdomains? (4) Do supervisors apply the rating scale on the rubric in the intended manner? (5) What differences exist between faculty, field experience supervisors, and classroom teachers in the application of the rubric to specific teaching subdomains?