Introduction: The proposed poster will describe the observational data collection methodology and preliminary findings from a large-scale efficacy study examining a corequisite mathematics intervention at the community college level. Specifically, the larger study is conducting a randomized controlled trial of the Carnegie Math Pathways (CMP) Statway College with Corequisite program as compared to a business-as-usual statistics corequisite course at Dallas County Community College. This research is funded by the Institute of Education Sciences (IES). The participating institution, Dallas College, is one of the largest colleges in Texas, serving over 100,000 students annually. The observation component of this study uses a new, highly validated protocol: the Toolkit for Assessing Mathematics Instruction -- Observation Protocol (TAMI-OP). The TAMI-OP was used to conduct informative classroom observations and to build narratives of both the treatment and comparison classrooms. We share our experiences to provide knowledge to fellow researchers, educators, and policymakers in measuring program fidelity and observational best practices using the TAMI-OP tool. Background/Context: Mathematics, specifically remedial or developmental mathematics, has been identified as one of the largest barriers to community college degree completion. An estimated 80% of students who start in developmental mathematics do not complete any college-level math course within 3 years (Bailey, Jeong, & Cho, 2010). To scale this percentage, approximately 1.7 million first-time undergraduate students enroll in developmental math courses each year. Students of color and poverty are overrepresented in this population (Complete College America [CCA], 2012). In response, the Statway College with Corequisite program introduces a statistics corequisite course which embeds innovative math problems, a collaborative learning model, and socio-emotional support for students into the curricula, as supported by research-based pedagogy (Strother & Klipple, 2019; CCA, 2012; Edwards & Beattie, 2016). Conducting classroom observations of Statway and non-Statway classrooms each semester using the TAMI-OP facilitates our evaluation of the fidelity and impact of implementation of the program. Data Collection: Selection and Use of the TAMI-OP: The TAMI-OP is an observational tool designed specifically for observing mathematics classrooms, adapted in part from existing classroom observation protocols (i.e., TDOP: Teaching Dimensions Observational Protocol, COPUS: Classroom Observation Protocol for Undergraduate STEM). The TAMI-OP is inherently descriptive rather than evaluative; it intends to describe and report practices used in mathematics classrooms without consideration of the effectiveness or implementation expertise of these practices (Hayward, Laursen, & Weston, 2017). Trained observers record classroom behaviors in two-minute intervals, noting the frequency and type of student and teacher activities and interactions. Data is entered in real time into an Excel spreadsheet template with pre-determined behavioral categories. The TAMI-OP additionally offers several post-observation qualitative items for holistic reflection (Chi & Wylie, 2014). The descriptive nature of the TAMI-OP facilitates evaluation of the implementation fidelity of the Statway corequisite program. For instance, the TAMI-OP quantifies the time students spend engaged in structured and spontaneous group work (WG, OG), individual work (Ind), and direct instruction (Lec). These measurements can be used to gauge the presence of the active, collaborative learning model which is characteristic of the Statway program. Additionally, the TAMI-OP records each time a student answers a question with justification or reasoning (AnRs), which can help provide an estimate of student conceptual understanding. The TAMI-OP also builds a narrative of student engagement by incorporating the ICAP scale, which categorizes student involvement as either "Interactive, Constructive, Active," or "Passive" within each two-minute segment. In Fall 2022, fifteen classrooms (seven Statway and eight non-Statway classrooms) were observed, for approximately 80 minutes each. This spring, an additional seven classrooms (three Statway and four non-Statway classrooms) were observed, for the same time duration. In the summer, we plan to conduct two or more supplemental observations. The number of observations is contingent upon the number of active, face-to-face sections. Advantages, Challenges, and Recommendations: Advantages: The two-minute segmentation of the TAMI-OP offers a granular glimpse into student and teacher classroom behaviors. Interested parties can view a detailed record of classroom activity without having attended the observation live. Additionally, observers may calibrate interrater reliability over a short sequence of training sessions, as compared to highly evaluative protocols which often require extensive training and calibration. The TAMI-OP is currently free. Challenges: The temporal precision of the TAMI-OP requires observers to remain acutely focused throughout the observation, with even brief inattention or distraction resulting in missing data. As the TAMI-OP does not report instructional practice effectiveness, some observation teams may prefer more evaluative protocols. Recommendations: We advise prospective observers to discuss TAMI-OP items with team members to ensure shared understanding. We found approximately five rounds of interrater reliability calibration to be sufficient. Observers may also discuss which items are likely to be indicative of implementation fidelity. If possible, we suggest piloting usage of the protocol in a face-to-face classroom prior to data collection. Preliminary Findings: Our TAMI-OP data from Fall 2022 reveal that Statway teachers spent less time engaged in direct instruction (i.e., lecturing to students, or Lec) than their non-Statway peers. Students in Statway classrooms additionally spent more time involved in structured group work and individual work (WG, Ind). These findings are aligned with the collaborative Statway pedagogy, which encourages increased student-to-student interaction. Alternatively, students in non-Statway classrooms spent more time in spontaneous group work (OG). The TAMI-OP also revealed that students in Statway classrooms answered more questions with justification or reasoning (AnRs). However, it was simultaneously found that teachers in non-Statway classrooms more frequently asked students reasoning-invoking questions (QRs). It is noteworthy that the total number of reasoning-invoking questions across classrooms was low. Conclusion and Next Steps: Our next steps include continued analysis of the Fall 2022 data and data collection in the spring and summer of 2023. Further results of Fall 2022 data analysis will be available for the conference, as well as preliminary findings from Spring 2023 and Summer 2023. We propose use of the TAMI-OP as an accessible and informative protocol for observing mathematics classrooms, and share its success in observing the fidelity of implementation of Statway College with Corequisite and the comparison curriculum as part of our larger efficacy study.