Typically, departments and universities evaluate and/or provide feedback to college mathematics instructors on their teaching by conducting classroom observations. When these observations are guided by an observation protocol, the protocol provides a particular lens that focuses the observer's attention on certain aspects of instruction. To understand the way in which mathematics departments are currently evaluating teaching and investigate whether they align with inquiry-based mathematics education practices, we evaluated 24 observation protocols from universities around the United States. Through our analysis we found that protocols mainly include closed-response items that are evaluative, focus more on the instructor than the students or the content, and rarely attend to inquiry-based mathematics education practices. These results suggest that where we aspire the field to go and where it is currently at requires significant and targeted work to achieve. Moreover, our findings provide suggestions and have implications for both those who develop observation protocols and those who use observation protocols as a measure of teaching practices. Our study highlights the need for careful explicit and clear purposes when observations are used for performance measures.