NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED656802
Record Type: Non-Journal
Publication Date: 2021-Sep-27
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Baseline Equivalency Measures in High Dimensions
Mark Fredrickson; Ben B. Hansen
Society for Research on Educational Effectiveness
Context: Assessments of baseline equivalency of intervention and control groups, "balance," play a critical role in evaluating educational interventions. The highest What Works Clearinghouse (WWC) of the Institute of Educational Studies (IES) standard for educational studies, "Meets WWC Design Standards Without Reservations," is reserved for randomized controlled trials (RCTs) without substantial attrition (IES, 2017), though specific subject matter protocols may also require demonstration of baseline equivalency on the primary outcome measures (e.g., IES, 2018). Randomized controlled trials with high rates of attrition and observational quasi-experimental designs may receive the "Meets WWC Design Standards With Reservations" designation if they able to demonstrate balance on critical baseline variables. Balance assessments are also useful in study design. "Restricted randomization" limits possible treatment regimens to those with no more than a given amount of imbalance (Morgan & Rubin, 2012). A common choice to quantify baseline equivalency is the Mahalanobis distance between treated and control group means (Hansen & Bowers, 2008; Morgan & Rubin, 2012): M = (n_1 n_0)/(n_1 + n_0) (X_1 - X_0)' Sigma^{-1} (X_1 - X_0), where n_j and X_j are the number and means of the treatment (j = 1) and control (j = 0) assignment units. Here Sigma^{-1} is an inverted sample covariance matrix for the baseline variables. When units are blocked into strata, a natural extension uses within stratum differences and a scaling matrix composed of stratum level covariances. In asymptotics with a fixed number of covariates and the number of assignment units tending to infinity, M has a well-behaved distributional limit (Li et al., 2018), ordinarily X[superscript 2] on rank([sigma]) degrees of freedom (Hansen & Bowers, 2008). Insofar as high dimensional balance assessment has been considered, previous authors suggest the use of a generalized inverse or alternative covariance matrix instead of [sigma superscript -1] to solve the numerical problems, but fail to consider that the usual limiting distribution may no longer apply (Branson & Shao, 2021; Hansen & Bowers, 2008; Morgan & Rubin, 2012). Research Questions: Our first research question is, "Does the usual asymptotic approximation of M using a X[superscript 2] distribution hold when the number of baseline variables approaches or exceeds the number of assignment units?" The number of variables in educational interventions can be quite large relative to the number of assignment units, particularly for cluster randomized trials. We investigate the quality of the usual approximation to the true distribution of M. If the usual approximation proves inadequate, our next research question is, "Can the true distribution of M be better approximated using higher moments of the distribution?" The asymptotic approximation based on a fixed number of variables uses only the first moment of the distribution. The second moment may provide useful information on the shape of the true distribution, allowing for more accurate approximations. Our third research questions asks, "Are there alternative statistics that provide better balance assessments when the number of variables is large?" Modifications of M or alternative measures of discrepancy between the treated and control groups may be preferable in the high-dimensional setting. Setting: We re-analyze a school based RCT reported by Gamoran et al. (2012) fielded in Phoenix, Arizona and Austin, Texas. To simulate the setting of researchers evaluating balance within a single state, we focus on the schools in Phoenix, AZ. Subjects: 1477 first grade children and their families recruited at 26 participating schools in Phoenix, AZ. Intervention: Within treatment schools, families were encouraged to attend events based on the FAST social capital building curriculum. Gamoran et al. (2012) gives additional details. Research Design: Schools were located in three Phoenix area districts (with six, eight, and twelve participating schools). Within each district, half of the schools were assigned to treatment. Table 1 gives the variables included in the data release. The data were released with categorical variables to ensure anonymity of subjects. To reflect the clustered nature of the design, we aggregate student categories to get counts per school and apply the principal components to create 50 orthogonal variables. Analysis: To establish the true distribution of the balance statistic M, we generated 10,000 treatment allocations. Figure 1 presents the empirical distribution of M as the number of variables increases and shows that the distribution tends to cluster near its mean more closely than the theoretical X[superscript 2] distribution. For 23 or more variables, the number of schools less the number of blocks, the distribution becomes a constant, an analytical result we have proved for any set of background variables. Using closed form calculations for the variance of M, we propose a moment matching correction Satterthwaite (1946) with the same mean and variance of M. Figure 1 performs well, even when the number of background variables is large. While the Satterthwaite correction represents a distinct improvement at higher variable counts, this correction suffers as well as the statistic approaches degeneracy. By the Eckart-Young-Mirsky theorem, the best rank-k approximation to a matrix is constructed from the k left and right singular vectors. We select k to maximize the variance of M, similar to recommendations from the principal components analysis literature. Figure 2 shows the true variance of M as the number of principal components increases, suggesting that using 12 principal components would be a reasonable choice for these data. Conclusions: Our first research question asked if the usual limiting distribution of M holds in typical educational RCTs, such as those funded by the IES. We prove that the Mahalanobis statistic will always degenerate when the number of variables exceeds the number of units less the number of strata. For our second research question, we derived variance calculations for M. We found better approximations using a correction that matches both the mean and variance of M. Researchers may also prefer to use Monte Carlo techniques, as demonstrated in Figure 1, to estimate the distribution of M more precisely. For our final research question, we suggest using a rank reduced approximation, selected to have the maximum variance. We continue to explore alternative statistics, particularly those that use a different scaling matrix.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Education; Early Childhood Education; Grade 1; Primary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Arizona (Phoenix); Texas (Austin)
Grant or Contract Numbers: N/A