The COVID-19 pandemic caused tremendous upheaval in schooling. In addition to its devasting effects on students' academic development, the disruptions to schooling had important consequences for researchers conducting effectiveness studies on educational programs during this era. Given the likelihood of future large-scale disruptions, it is important for researchers to plan resilient studies and think critically about possible adaptations when such turbulence arises. In this article, we utilize qualitative case study analysis to examine how researchers evaluating educational programs in the pandemic period adjusted to turbulent conditions through design pivots to ensure the feasibility of research. We find that researchers struggled to strike a balance between the evaluations that were intended and those that could realistically be accomplished. We identify how the challenges of the pandemic period and design pivots raised potential threats to validity, illuminate some promising practices that arose during the pandemic period, and provide recommendations for future research and evaluation programs focused on studying the effectiveness of educational programs during times of profound disruption.