In this study, we investigated factors that influence raters' application of the scoring criteria of an Explicit Instruction (EI) observation protocol using many-faceted Rasch measurement (MFRM) and think aloud analysis. Specifically, we investigated the extent to which raters are able to consistently represent the scoring criteria in the EI rubric, how raters discriminate among levels of performance on each instructional element, and the consistency with which the raters applied evidence to support their scoring decisions. Video observations of instruction from 30 special education teachers across three states were collected. External raters (n = 15) observed and scored videos, providing rationales for item level scores. MFRM analyses showed that raters differed in their severity, but each individual rater was able to apply scoring criteria in a consistent manner. Think aloud analyses showed that raters varied in how they interpreted items and in the evidence they used to support their scores. Implications for research are discussed. [This paper was published in "Studies in Educational Evaluation" v64 Article 100827 2020.]