Evaluation designs seek to provide evidence of impact and collect formative data to understand and improve programs and inform decisions. Formative data is commonly used at the onset of developing a new program to modify and improve before full implementation. However, formative data is often collected throughout the life of the program to shape…

Introduction: Missing data are bound to occur in education intervention research. Reasons vary but always have the consequence of reducing sample sizes and can, under certain conditions, seriously bias estimated intervention effects. A wide array of methods have been developed for handling missing data (Enders, 2023). Whereas older approaches…

We consider a class of multiple-group individually-randomized group trials (IRGTs) that introduces a (partially) cross-classified structure in the treatment condition (only). The novel feature of this design is that the nature of the treatment induces a clustering structure that involves two or more non-nested groups among individuals in the…

Background: In educational settings, individuals are often best served by an intervention that is adapted over sequential stages to suit their initial and changing needs. The salience of an adaptive intervention is, perhaps, most clear in the classroom. Learning itself is a sequential process: mastering a given concept or technique frequently…

Background: Power analyses for a cluster-randomized trial (CRT) require estimates of additional design parameters beyond those needed for an individually randomized trial. In a 2-level CRT, there are two sample sizes, the number of clusters and the number of individuals per cluster. The intraclass correlation (ICC), or the proportion of variance…

Background: The generalizability of a study's results continues to be at the forefront of concerns in evaluation research in education (Tipton & Olsen, 2018). Over the past decade, statisticians have developed methods, mainly based on propensity scores, to improve generalizations in the absence of random sampling (Stuart et al., 2011; Tipton,…

Background/Context: The reckoning with racial injustice and growing inequality that have become hallmarks of the early 2020s in the United States has implications for impact analysis and the evidence it produced for public policy decision-making. Various researchers have highlighted the shortcomings that impact analyses have when estimating the…

Background: Evidence from education randomized controlled trials (RCTs) in low- and middle-income countries (LMICs) demonstrates how interventions can improve children's educational achievement [1, 2, 3, 4]. RCTs assess the impact of an intervention by comparing outcomes--aggregate test scores--between treatment and control groups. A review of…

I. Purpose of the Study: Detecting whether interventions work or not (through main effect analysis) can provide empirical evidence regarding the causal linkage between malleable factors (e.g., interventions) and learner outcomes. In complement, moderation analyses help delineate for whom and under what conditions intervention effects are most…

Background: The analysis of mediating mechanisms attempts to examine the impact of, for example, teacher training by dismantling and testing the core components that comprise an underlying theory and program. This process is seen as a critical step in evaluating programs and mapping teacher development because it provides empirical tests of each…

Background: Education scientists are increasingly interested in constructing interventions that are adaptive over time to suit the evolving needs of students, classrooms, or schools. Such "adaptive interventions" (also referred to as dynamic treatment regimens or dynamic instructional regimes) determine which treatment should be offered…

Random encouragement designs are randomized controlled trials (RCTs) that test interventions aimed at increasing participation in a program or activity whose take up is not universal. In these RCTs, instead of randomizing individuals or clusters directly into treatment and control groups to participate in a program or activity, the randomization…

Background and Context: Every student is a powerful learner capable of succeeding in mathematics. The EF+Math Program aims to dramatically improve math outcomes for Black and Latinx students and students of all races experiencing poverty (hereafter, our Priority Students). We are a community of researchers, developers, educators, and students who…

Context: A perennial problem in applied statistics is the inability to justify strong claims about cause-and-effect relationships without full knowledge of the mechanism determining selection into treatment. Few research designs other than the well-implemented random assignment study meet this requirement. Researchers have proposed partial…

Background: Education researchers typically estimate average program effects with regression; if they are interested in heterogeneous effects, they include an interaction in the model. Such models quantify and infer the influences of each covariate on the effect via interaction coefficients and their associated p-values or confidence intervals.…