Over the past 15 years, the education research community has advocated for rigorous research designs that support causal inferences, for research that provides more generalizable results across settings, and for the value of research-practice partnerships that inform the design of local programs and policies. We propose the multi-cohort, longitudinal experiment (MCLE) as one approach to balancing these three, sometimes competing goals in a single study. We describe our application of an MCLE design to evaluate a teacher coaching program, where we find that changes in program features related to personnel, content, and duration coincided with substantial differences in effectiveness across three cohorts of the experiment. Our analyses and corresponding recommendations can help researchers weigh the benefits and trade-offs of the MCLE design.