This project involved examining the most widely used instrument designed to evaluate the quality of early learning environments, the Early Childhood Environment Rating Scale-Revised Edition (ECERS-R). There are many aspects related to the way that the ECERS-R is used in practice that can vary from one observation to the next. The method in which the ECERS-R is scored, the time point of the year in which the ECERS-R observations are conducted, the length of an observation period, and the start time of an observation are dependent entirely by the observer. However, there is little research on the susceptibility of the resulting scores to the influence of such measurement methods. Although the instrument developers can't possibly regulate exactly how the ECERS-R is used, it is important to know what the implications of these alternatives are. Using secondary data from over 250 classrooms, this article demonstrates how the effects of these measurement attributes can have important considerations for the policy realm, particularly when ECERS-R scores are tied to program funding considerations. (Contains 8 tables and 1 figure.)