ERIC Number: ED663246
Record Type: Non-Journal
Publication Date: 2024-Sep-18
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Heterogeneity Estimation in Meta-Analysis: Investigating Methods for Dependent Effect Size Estimates
Jingru Zhang; James E. Pustejovsky
Society for Research on Educational Effectiveness
Background/Context: In meta-analysis examining educational intervention, characterizing heterogeneity and exploring the sources of variation in synthesized effects have become increasingly prominent areas of interest. When combining results from a collection of studies, statistical dependency among their effects size estimates will arise when a single study provides multiple effect sizes or when studies use similar operational procedures across independent samples. Robust variance estimation (RVE) methods offer a viable approach to incorporate multiple dependent effect sizes into a single meta-regression model, even in the presence of unknown dependency structures. Within the RVE framework, several working models have been proposed. Currently widely used working models include the correlated effects (CE) model and the hierarchical effects (HE) model (Hedges et al., 2010), which differ in their emphasis of the dependency structure. Pustejovsky and Tipton (2022) proposed the correlated-and-hierarchical effects (CHE) model that encompasses both types of dependence, thus avoiding the need to choose between working models when neither option adequately describes the structure of the data. Other working models used in practice include the aggregated effects (AE) model (Pustejovsky and Chen, 2024), independent effects (IE) model (Becker, 2000), and multi-level meta-analysis (MLMA) model (Van den Noortgate et al., 2013, 2015). Objectives: As these working models become increasingly integrated into meta-analysis of dependent ES, there is a need for guidance about how to select an appropriate working model and, in particular, how the heterogeneity estimators developed for different working models can be interpreted. To develop such guidance, we focused here on the properties and performance of the heterogeneity estimators proposed for the CE, HE, MLMA, and other working models under conditions where they may be mis-specified relative to the true dependence structure. Beginning with the general correlated-and-hierarchical effects (CHE) model, we derive closed-form expressions for the expectation of the heterogeneity estimators. We then utilize these expressions to provide insights into how these estimators can be interpreted in contexts where heterogeneity may be expected at both the between- and within-study levels. Theoretical Framework: Consider a sample of ES estimates consisting of K estimates that have been collected by first sampling J studies, then sampling k[subscript j] [greater than or equal to] 1 estimates within the j[superscript th] study, for j = 1,...,J. Let T[subscript ij] denote ES estimate i from study j, which is an unbiased estimate of an ES parameter [theta][subscript ij], with corresponding fixed and known standard error [sigma][subscript ij], for i = 1...k[subscript j] and j = 1...J. The Correlated and Hierarchical Effects (CHE) model combines both correlated sampling errors and hierarchical dependence structures into a structure with several random effects. Specifically, T = x[subscript ij][beta] + u[subscript j] + v[subscript ij] + e[subscript ij], where Var(u[subscript j]) = [tau][superscript 2], Var(v[subscript ij]) = [omega][superscript 2], and Var(e[subscript ij]) = [sigma][superscript 2][subscript ij]. For simplicity, we will assume that sampling variances are constant (or nearly constant) within each study, so that [sigma][superscript 2][subscript ij] [approximately equal to] [sigma][superscript 2][subscript j] and that there is a constant correlation between effect sizes within a study, so that Cov(e[subscript hj], e[subscript ij]) = [rho][sigma][superscript 2][subscript j]. The cluster-specific covariance matrices [phi][subscript j] implied by the CHE working model include a between-study variance component [tau][superscript 2], a within-study variance component [omega][superscript 2], and correlation [rho] between pairs of ES from the same study. Other working models can be seen as special cases of the CHE model. Table 1 shows the covariance structure, variance components estimators, and their estimation methods for six working models that have been proposed for meta-analysis of dependent ES. Table 2 summarizes the expectation of each estimator under the CHE model. These expressions can be interpreted as showing the extent to which the estimators derived from each working model deviate from the true parameter, given the assumption that the actual data generation model adheres to the CHE structure. Empirical Example: To illustrate how the choice of different working models affects variance components and meta-regression coefficient estimation, we empirically reanalyzed the work by Assink and colleagues (2015) which reported a meta-analysis examining the association between mental health disorders of juveniles and juvenile offender recidivism. In this example, we used a subset of the data with 100 effect sizes from 17 studies. Within each study, there are between 1 and 22 effect size estimates. Studies with multiple effect sizes measured multiple outcomes, and there are possible non-zero sampling correlations among effect sizes. Figure 1 shows the allocation of variances at different levels when applying different working models. Three main messages come out from this figure. First, there is a variation among the total heterogeneity estimates across different working models. Second, the models that estimate both between-study and within-study heterogeneity, the CHE, MLMA, and HE models, differ in terms of how they attribute the proportion of between-study and within-study variance. Third, for both within and between study variance components, there is a variation in estimation across working models. Figure 2 shows the comparison of fixed effects estimates for the working models. The overall average effect sizes are statistically distinguishable from zero based on the results from all models, but we still see a difference in the point estimates and confidence interval. The differences in point estimates illustrate a downstream consequence of the choice of working model for inferences based on meta-regression results. Conclusion: In the current study, we have illustrated how the choice working model can influence not only the precision of average ES and moderator effects but also the findings regarding the degree of overall heterogeneity of effects and the structure of the heterogeneity. Choosing a flexible working model such as CHE offers benefits in terms of better capturing the types of dependence that occur in practice. As educational research studies are increasingly diversify in their use of technology, cultural context, and impact on policy, meta-analytical strategies to synthesize these studies will benefit from tools that can effectively capture and describe the different levels of variation in effect sizes. In future work, we would like to investigate further on heterogeneity estimator performance under other complex data generation process, include a structure with additional levels of nesting that is regularly seen in practice when conducting large-scale systematic reviews.
Descriptors: Meta Analysis, Investigations, Effect Size, Computation, Regression (Statistics), Hierarchical Linear Modeling
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A