NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED659432
Record Type: Non-Journal
Publication Date: 2023-Sep-29
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Mapping the Landscape of the Empirical Literature on Power Analysis Design Parameters
Joseph Taylor; Dung Pham; Paige Whitney; Jonathan Hood; Lamech Mbise; Qi Zhang; Jessaca Spybrook
Society for Research on Educational Effectiveness
Background: Power analyses for a cluster-randomized trial (CRT) require estimates of additional design parameters beyond those needed for an individually randomized trial. In a 2-level CRT, there are two sample sizes, the number of clusters and the number of individuals per cluster. The intraclass correlation (ICC), or the proportion of variance in the outcomes that is among clusters, is also needed. Lastly, in a 2-level CRT, covariates (e.g. pretest, demographic characteristics) can explain variance in the outcome at the individual level or the cluster level, hence there are often two estimates of the variance explained (R2). In order to equip researchers planning CRTs with the necessary tools to conduct accurate power analyses, the methodological literature began to expand quickly with empirical studies that estimated power analysis design parameters from national and state-level databases (e.g., Hedges & Hedberg, 2007; Schochet, 2008). However, it is now clear that there is no one paper/resource that will cover the breadth of designs being proposed and conducted in education. This literature base is now dispersed across numerous journals in education and related fields and now spans nearly two decades. This can make it challenging for researchers planning CRTs to identify the most relevant empirical estimates of design parameters for a particular study design. Purpose: In response to the current dispersed nature of the power analysis literature, the purpose of this paper is to map the landscape of recent studies that have estimated power analysis parameters. In doing so, we catalog the characteristics of research studies that have derived such estimates with the intent of illuminating areas where significant work has been done and where gaps in the literature might be filled by future research. Setting: NA. Population/Participants/Subjects: NA. Intervention/Program/Practice: NA. Research Design: Systematic review methods were applied to the search, selection, and coding of studies. These methods were conducted and reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 checklist and flow diagram for reporting in systematic reviews and meta-analyses (see http://www.prisma-statement.org). Data Collection and Analysis: Collection: We used a comprehensive search strategy to identify and retrieve all relevant studies. Approximately 300 databases were searched including large databases of indexed research in education and social sciences such as Education Resources Information Center (ERIC), PsychINFO, Education Research Complete, and Academic Search Premier. The following search terms were used to locate relevant studies: "design parameters" AND "statistical power" AND "education". The following two filters were applied to the search: Peer-Reviewed and Full Text Online. In addition, the search was limited to the following disciplines: Education, mathematics, statistics, sociology, social welfare, social science, and psychology. The search retrieved 87 total articles to which we applied our eligibility criteria. Eligibility studies: i) provided empirical estimates of design parameters necessary for statistical power calculation for quantitative research designs in education research, ii) estimated the parameters using secondary data such as data from funded impact studies or district, state, national, or international data sets, iii) were published in a peer-reviewed journal, and iv) were published in English. Studies were excluded if they did not provide an empirical estimate of a design parameter necessary for calculating power for a multilevel study in education research. These included simulation studies, studies providing only a theoretical or conceptual framework, and literature review studies. After full text eligibility screening, 25 studies were included in the synthesis. For these 25 studies, the following characteristics (primary coding variables) were extracted: a) the impact designs they targeted/informed (e.g., provided ICC estimates to inform two-level CRT designs), b) the subject domains of the outcome measures (e.g., reading or math achievement, c) the age of the persons from which the outcomes were collected, d) the source of the data (e.g., local evaluation, nationally representative test data), e) the country of origin for the data, and f) the types of covariates tested in R2 analyses. Analysis: Frequency tabulation and Evidence and Gap Maps (EGMs) were used to tabulate and visually display the results of the evidence synthesis. Absolute and relative frequency for each level of the primary coding variables was computed. The frequency statistics provided the input for the EGMs, which provide a visual presentation of the existing evidence and gaps on selected combinations of the coding variables. Results: Due to word restrictions for this abstract, we present here a sample set of findings and an associated EGM. These findings pertain to the designs that each study in the literature targeted or sought to inform. The conference presentation, if accepted, will present all results and EGMs. As seen in Figure 1, we found 22 out of 25 studies (88%) reported empirical estimates for design parameters for CRTs, of which two studies also provided parameter estimates for randomized block designs (specifically, MSCRTs). The remaining studies (n = 3 or 12%) provided empirical estimates for design parameters specific to randomized block designs (e.g. MSTs and MSCRTs). Clearly, there is limited research that provide design parameter estimates that inform the planning of randomized block designs relative to CRTs. Referencing our EGM in Figure 2, where the size of the circle is proportional to the number of relevant studies, our findings show that the empirical design parameters literature informs two-level and three-level CRT designs to a much greater extent than four-level CRT designs. Further, this literature focuses proportionally more on informing the design of impact studies with student outcomes. Conclusions: Additional findings that will be presented if the paper is accepted include clear evidence gaps in the empirical literature for post-secondary outcomes, non-cognitive outcomes, outcomes from non-US sources, and effect size heterogeneity. We hope that by identifying these gaps, we can spur interest in research in these areas, both by the investigators themselves and by funding agencies that value the scientific rigor and cost-management benefits that are afforded by accurate power analyses.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Information Analyses
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A