ERIC Number: ED658680
Record Type: Non-Journal
Publication Date: 2022-Sep-23
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Missing, Presumed Different: Quantifying the Risk of Attrition Bias in Education Evaluations
Luke Miratrix; Ben Weidmann
Society for Research on Educational Effectiveness
Background/Context: Attrition has been described as "the Achilles Heel of the randomized experiment" (Shadish et al., 1998 p.3). Attrition looms as a threat because it can undermine group equivalence, eroding the methodological strength at the heart of a randomized evaluation. In particular, attrition could result in unobserved differences between treatment arms, leading to biased estimates of treatment impact. But how large are these biases in practice? Purpose/Objective/Research Question: Our paper makes four contributions: (1) Estimate attrition bias for 10 education RCTs, spanning 22 outcomes. We are able to do this by accessing census data that include attainment information from those pupils who attrited from the 10 RCTs. Using techniques from meta-analysis, we then estimate the typical magnitude of attrition bias across our studies and outcomes; (2) Present a framework for decomposing attrition bias. We illustrate that attrition bias is a function of four components: the rate of attrition and association between attrition and outcome in each treatment arm. We quantify the magnitude of these components; (3) Examine the "Missing At Random" (MAR) assumption in our context; and (4) Provide two practical recommendations for researchers: i) incorporate uncertainty from attrition bias and ii) check whether conclusions are sensitive to 'worst-observed case' attrition mechanisms. Setting and Data: Our analysis relies on a unique set of linked databases in England. The key data source is an archive of RCTs that can be linked to the National Pupil Database (NPD). The NPD is a census of publicly funded schools and pupils that provides testing outcomes for all students, even those who attrited from the initial studies. We examine 10 interventions (see table 1 for description). The interventions were diverse in their approaches to raise academic achievement in literacy, numeracy and science, for pupils aged 6 to 15. Attrition ranged from less than 10% to more than 40%, with a mean of around 19%. Differential attrition was as high as 20%. We aligned each outcome in the original trial to a corresponding state-wide test; given the requirements for RCTs in our archive, we argue this correspondence is reasonable. Research Design: Consider Model 1 in which Y[subscript ijkw] is the outcome that student i in school j achieves in intervention k for outcome w, and T[subscript ij] is a binary treatment indicator: Y[subscript ijkw] = a[subscript j] + [tau]T[subscript ij] + e[subscript ij], a[subscript j] ~ N( a[subscript 0], [delta superscript 2 subscript a] ), e[subscript ij] ~ N( 0, [delta superscript 2 subscript e] ). Using this model, we estimate attrition bias for a given study and observation with three steps: (i) Fit Model 1 only using units from the "responder sample" (those students who provided data for the initial evaluation). This gives [tau]-hat[subscript R], the estimate of the ATE for the responder sample. (ii) Refit Model 1 using the "full sample," (all units, with valid pupil identifiers recorded as being randomized) obtaining [tau]-hat[subscript FULL], the estimated ATE for the full sample. (iii) Take the difference: [beta]-hat = [tau]-hat[subscript FULL]- [tau]-hat[subscript R]. This is an estimate of attrition bias, including additional uncertainty due to factors such as sampling variation, imbalance in random assignment, and measurement error. Covariate adjustment to repair attrition bias. We also estimate attrition bias after covariate adjustment by extending the above model to include baseline covariates. We denote estimates of bias "after" adjustment by [beta superscript X]-hat. There are two reasons for adjustment: first, we are interested in the magnitude of attrition bias "in practice" (all the RCTs in our sample used covariates in their analyses). Second, we can examine the extent to which adjustment repairs attrition bias. We find adjustment substantially reduces bias, i.e. |[beta]-hat| is typically larger than |[beta superscript X]-hat|. The distribution of attrition bias. Our primary interest is to look across trials and understand the "distribution" of attrition bias. Our empirical estimates will be over-dispersed as they are estimates of the bias; to take over-dispersion into account we first estimate standard errors for the bias estimates using a resampling approach, and then use tools from meta-analysis to adjust our bias estimates and estimate characteristics of the distribution of true bias. This analysis allows us to achieve two major goals: (1) Estimate the typical magnitude of attrition bias for our setting; and (2) Present estimated distributions for [beta superscript X]-hat and [beta superscript X]-hat that are not over-dispersed. Results: Our final estimates are on Figure 1. The estimated average attrition bias is -0.013ES units (effect size units), with mean absolute value of 0.026ES units. Once we control for covariates, including a pre-test, the estimated distribution of bias, post adjustment, has a mean of [nu] = -0.005ES units, and a mean absolute value of 0.014ES units. No values of attrition bias have a magnitude greater than 0.034ES units. To put the magnitude of these estimates into context, consider that the What Works Clearinghouse set a threshold for problematic bias at 0.05 ES units (WWC, 2014). Even the estimates that do "not" condition on covariates are below this threshold. We discuss connections to US contexts further in our complete work. We also model attrition as the differential relationship of mean outcome for the responder and full sample for each treatment arm to assess differential attrition in our studies. We find overall impact to be minimal, in line with our primary results. Conclusion: Overall, our analyses suggest that the threat of attrition bias is limited in our context. This is despite the fact that, in 7 out of the 10 RCTs we analyzed, the researchers themselves reported serious concerns about attrition as a risk. We argue our set of experiments is representative of school-level interventions, and also call for further research to see if these trends replicate in other contexts. This is not to say that attrition mechanisms can safely be treated as "missing at random." In our full results, we find evidence that students who leave studies tend to perform worse than those who remain. That said, these associations do not appear to be strong enough to induce large-scale bias. We suggest that researchers respond to this evidence by completing sensitivity analyses using empirically-grounded estimates of attrition mechanisms (such as the ones we present). We provide simple methods for doing this in our paper.
Descriptors: Educational Research, Statistical Bias, Attrition (Research Studies), Computation, Evidence, Effect Size
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A