A well-designed experiment is the best method for establishing efficacy of any intervention, be it medical, behavioral, or educational in nature. This paper reviews the steps necessary in conducting a comparative experiment in an educational setting, and illustrates how these steps might be fulfilled within the context of a large-scale randomized experiment conducted in an introductory statistics course. The primary goal of this paper is to help researchers identify salient issues to consider and potential pitfalls to avoid when designing a comparative experiment in an educational setting.