This paper aims to describe the development process of the Observation Protocol for Teaching Activities in Mathematics (POAEM) and to report the findings from the qualitative and statistical analyses used to provide evidence of validity and reliability of the information collected with the first version of the POAEM. As part of this development process, 20 teachers from Mexican primary schools were videotaped twice while teaching mathematics. The study assessed the reliability of the POAEM rubrics. Results showed that the dimensional structure of the instrument can be grouped in one factor. A generalizability study provided information on the different sources of error in the measurement, showing that the dimensions accounted for 78% of the variance. This study provides an exemplar of the design and validation of an instrument that can help other researchers develop their own instruments and data collection to generate evidence of validity and reliability in different sociocultural contexts.