This paper discusses the challenges and lessons learned from conducting observations to measure the quality of classroom practice for a large-scale study of elementary teachers' mathematics instruction. Specifically, this paper shares our process for obtaining valid data for quality of elementary mathematics instruction; what we learned can inform similar efforts by large, interdisciplinary teams, as well as by instructional coaches and evaluators. Considerations for selecting and testing the viability of a classroom observation instrument, as well as the importance of conducting inter-rater reliability throughout the data collection process, are discussed.