End-of-semester evaluations provide scalable data for university administrators, but typically do not provide instructors with timely feedback to inform their teaching practices. Midsemester evaluations have the potential to provide instructors with beneficial formative feedback that can contribute to improved teaching practices and student engagement. However, existing research on the construction of valid, reliable midsemester tools is rare, and there are no existing midsemester evaluation scales that were constructed using education research and psychometric analysis. To address this gap, we designed and piloted a midsemester evaluation of teaching with 29 instructors and 1,350 undergraduate students. We found evidence that our Mid-Semester Evaluation of College Teaching (MSECT) is a valid and reliable measure of four constructs of effective teaching: classroom climate, content, teaching practices, and assessment. Furthermore, our factor structure remained consistent across instructor genders, providing evidence that the MSECT may be less susceptible to gender bias than prior student evaluation measures.