This article engages with the unsettled question of whether student response systems increase engagement in the classroom. The reported study used a multi-classroom, multi-disciplinary pilot deployment of the app-based student engagement platform Nearpod to test whether greater use of the technology yields greater student engagement. The study results were decidedly mixed, though they prompt a discussion of the construct validity of student engagement measures, particularly in a classroom with app-based platforms that reduce traditional signs of engagement (for example, raising hands). The results also have broader implications for engagement theory and call for problematizing the resulting questions being asked.