NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED663508
Record Type: Non-Journal
Publication Date: 2024-Sep-20
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Power Analyses for Estimation of Complier Average Causal Effects under Random Encouragement Designs in Education Research: Theory and Guidance
Peter Schochet
Society for Research on Educational Effectiveness
Random encouragement designs are randomized controlled trials (RCTs) that test interventions aimed at increasing participation in a program or activity whose take up is not universal. In these RCTs, instead of randomizing individuals or clusters directly into treatment and control groups to participate in a program or activity, the randomization involves forming treatment and control groups that are either encouraged, or not, to participate. The encouragement, for example, could provide information on the benefits of an action or reminders (nudges) to participate, where intervention features are often grounded in principles of behavioral science (Thaler & Sunstein, 2008; Hummel & Maedche, 2019). Random encouragement designs testing behavioral interventions are increasingly being used in education research to test a variety of interventions targeting pre-K to college students (Castleman et al., 2017). For example, Bird et al. (2021) recently tested a text messaging campaign to encourage students to apply for college financial aid, and Bergman (2021) randomly selected parents to receive additional information about their children's academic progress to encourage parental involvement in their children's schoolwork. The random encouragement design allows for unbiased estimation of the average treatment effect (ATE) of the encouragement (the "treatment") on both (i) participation in the targeted action (the "mediator") and (ii) longer-term "outcomes," such as student achievement. However, interest may also lie in estimating the causal mediating effects of participation itself on outcomes. An analytic complication with these analyses, however, is that participation decisions are likely to be correlated with unobserved factors affecting longer-term outcomes, which could lead to bias. Several estimation and design strategies have been proposed to overcome these sample selection biases to identify well-defined causal mediating estimands under plausible assumptions (e.g., Imai et al., 2013). Thus, the encouragement RCT is attractive in that it allows for rigorous estimation of the effects of participation in a program or activity when direct randomization is not feasible or ethical (such as randomizing parental involvement with their children's schoolwork). Further, even if direct randomization is practical (e.g., to an oversubscribed afterschool program), an encouragement design may still be preferred as it avoids the direct denial of program services to the control group, thereby reducing study burden and facilitating study recruitment and implementation. The viability of the random encouragement design for estimating mediating effects, however, depends on its statistical power. Accordingly, this work addresses the following research question: What are required sample sizes for the random encouragement design to yield sufficient power for detecting the effects of participation in a program or activity on participants' longer-term outcomes? We consider statistical power for the complier average causal effect (CACE) estimator, where participation is measured as binary, for behavioral encouragements (such as nudges) that are increasingly being tested in education trials. The CACE estimand pertains to the subgroup of compliers in the study population who would participate in the action in the treatment but not control condition (i.e., those induced to act by the encouragement). The CACE estimator is an instrumental variables (IV) estimator, where outcomes are modeled as a linear function of participation and treatment status is used as an instrument for participation (e.g., Abadie, 2002; Angrist et al., 1996; Imai et al., 2013; Imbens & Rubin, 2015; Sobel, 2008). We focus on the CACE estimand for several reasons. First, the What Works Clearinghouse (WWC) has set evidence standards for CACE analyses that are commonly used in practice (WWC, 2022), but not for other RCT estimation approaches for addressing noncompliance. Second, the CACE estimator has closed-form asymptotic variance formulas that facilitate power calculations. Finally, CACE analyses can be easily conducted using standard statistical packages. In this paper, we bring together the related literature to provide a unified approach for conducting power analyses for the CACE estimator for a broad range of education RCT designs. The approach uses a potential outcomes framework combined with standard monotonicity and exclusion restrictions for CACE identification (Angrist et al., 1996) that are likely to be plausible for many random encouragement designs conducted in education research. Key advantages of this approach are that it allows for treatment effect heterogeneity and makes no assumptions about the distribution of model error terms. The focus is on power for the "clustered" RCT design, where the behavioral encouragement is assigned to groups, which also covers (reduces to) the nonclustered design with individual-level randomization. We also consider extensions to the random block design. Because power analyses require variance estimators, we use generalized estimating equations (GEE) theory (Liang & Zeger, 1986) to obtain the asymptotic distribution of our considered CACE estimators with and without model covariates, thereby extending the results in Imbens and Angrist (1994) who only consider the nonclustered design without covariates. The resulting closed-form asymptotic variance formulas and normal distributions underlie our power formulas. Further, because the variance formulas contain several components that may not be known in advance, we consider various options for applying them in practice. Finally, to address our main research question, we conduct an illustrative power analysis using plausible parameter values to provide guidance on appropriate sample sizes for the CACE-based random encouragement design in education research. A Shiny R dashboard, Encourage_Power, is available at https://github.com/pschochet/Encourage_Power that conducts all the power analyses considered in the paper. Our illustrative power analysis shows that the random encouragement design requires very large samples to attain target CACE effects typically found in education RCTs. The key reason is that power for these RCTs is largely driven by the size of the ITT effects on compliance. Thus, these designs may only be viable if the behavioral encouragement has a meaningful effect on inducing participation, and in turn, if participation in the program or activity has a sizeable effect on improving longer-term outcomes. The availability of pre-treatment variables, highly predictive of the primary outcomes (such as pretests), may also be required to achieve target precision levels.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Related Records: EJ1457086
Publication Type: Reports - Evaluative
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A