It is well documented that classroom observation reports are used by school leaders as just one piece of the clinical supervision process of teachers (Cohen and Goldhaber, 2016). While the frequency of classroom observations might vary between school districts, recent regulations on Annual Professional Performance Review (APPR) have added more consistency to this process in New York State (USDOE, 2014). The challenge for school leaders has been to gain actionable data from observation reports in the school improvement process. Until recently, observation documents were filed as they occurred providing difficulty for school leaders to make connections between observations of individual teachers. Likewise, school district leaders were unable to elicit trends across classrooms without intentional, time consuming review of observation documents. The increasingly ubiquitous use of computer based software in the classroom observation process presents an opportunity for this analysis (Goldring et al., 2015). While most programs allow for quantitative analysis of observation scores and individual component ratings, none provide for the qualitative analysis of evidence collected by school administrators in the observation process. This paper attempts to initiate a process by which data from classroom observation reports can be systematically and efficiently analyzed through the use of available qualitative research software. The power of the qualitative research software to organize the data and allow the user to perform queries reveals limitless possibilities of analysis. School district leaders can use this information to develop instructional goals, train administrators how to document the occurrence of the instructional goals within classrooms, and then monitor progress on the implementation of those goals. Unlike other initiatives, this program of analysis capitalizes on a process that currently exists across all schools in New York State.