NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED663185
Record Type: Non-Journal
Publication Date: 2024-Sep-20
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Empirical Design Parameter Values for Planning Mediation Studies with Teacher and Student Literacy Outcomes
Amota Ataneka; Fangxing Bai; Yanli Xie; Ben Kelcey; Nianbo Dong; Leigh McLean; Geoffrey Phelps
Society for Research on Educational Effectiveness
Background: The analysis of mediating mechanisms attempts to examine the impact of, for example, teacher training by dismantling and testing the core components that comprise an underlying theory and program. This process is seen as a critical step in evaluating programs and mapping teacher development because it provides empirical tests of each component of a theory of action as well as how that theory functions as a coordinated system (Kelcey et al., 2017). However, there is scant empirical literature available to guide the effective and efficient design of mediation studies (Kelcey et al., 2020). Because the precision with which a study can identify mediation effects is tied to the mediator and outcome variance decompositions across levels, prospective design requires reasonable a priori estimates of parameters such as the variance partition coefficient (VPCs) or the intraclass correlation coefficients (ICCs) for the mediator and outcome. In developing power analyses, funding organization typically require a rationale or empirical basis for the assumed parameter values (e.g., IES RFA, 2023). Purpose: In this study, we contribute to this gap by developing an initial collection of empirical estimates that may serve as a practical starting point in the design of teacher mediation studies that concern with student outcomes in English Language and Arts (ELA). Our analyses draw on teacher and student variables within the context of a three-level structure: students nested in teachers nested in schools. Using three-level models, we estimated the VPCs of a broad range of teacher mediators and student outcomes as well as the proportion of variance explained (R2) at each level using commonly available covariates. Setting: Our analyses examine 31 teacher mediators and 5 student achievement outcomes across 3 datasets. For each mediator and outcome, we estimated variance decompositions (VPCs) and the predictive power of various covariates. Analysis: For unconditional decompositions of mediators, we modeled the mediator as follows: at the teacher-level, we used M[subscript jk] as the mediator value for teacher j in school k, [pi][subscript 0k] as the school-specific intercept and [epsilon][superscript M][subscript jk] the teacher-level error term. At the school-level, we modeled the school-specific intercepts as a function of the overall intercept ([zeta][subscript 00]) and school-level random effects (u[superscript M][subscript 0k]). The resulting model was [equations omitted]. In turn, we used the models to estimate the variance partition coefficients (VPCs) as [equation omitted]. A three-level model for the outcome. The unconditional model at the student-level, we used Y[subscript ijk] as the outcome for student i served by teacher j in school k, [beta][subscript 0jk] as the teacher-specific intercept and [epsilon][superscript Y][subscript ijk] as the student-level error term. At the teacher-level, we modeled the teacher-specific intercepts as a function of the school-specific intercepts ([gamma][subscript 00k]) and teacher-level random effects (u[superscript Y][subscript 0jk]). At the school-level, we modeled the school-specific intercepts as a function of the overall intercept ([zeta][subscript 0]) and school-level random effects (v[superscript Y][subscript 00k]). [equations omitted] We report the standardized VPCs for the teacher- ([rho][superscript L2][subscript Y]) and school-level ([rho][superscript L3][subscript Y]) as [equation omitted] and [equation omitted]. To estimate the proportion of variance explained by covariates, we built on the unconditional mediation models by introducing covariates at the appropriate level and their aggregates at hierarchical level(s). For a mediator model conditional on a teacher-level covariate (X), we have [equations omitted]. We introduced X[subscript 1jk] as a covariate for teacher j in school k with associated regression coefficient [pi][subscript 1] and [X-bar][subscript 1k] as its school-level aggregate with regression coefficient [zeta][subscript 01]. The residual variance components are now conditional upon the covariates. We estimated the proportion of variance explained by a covariate at the teacher- (R[superscript 2][subscript (M[subscript L2])) and school-levels (R[superscript 2][subscript (M[subscript L3])). for the mediator as [equation omitted] and [equation omitted]. For an outcome model conditional on a student-level covariate (Z), we have [equations omitted]. We estimated the proportion of variance explained by a covariate at the student- (R[superscript 2][subscript (Y[subscript L1])), teacher- (R[superscript 2][subscript (Y[subscript L2])), and school-level (R[superscript 2][subscript (Y[subscript L3])) as [equation omitted] and [equation omitted] and [equation omitted]. Data: The data used in this analysis comes from a rich collection of studies that draw on key teacher development constructs including the teacher knowledge, instruction and student achievement using high-quality and well-researched instruments. We outline and report on just two of the datasets for proposal brevity but our study includes a much larger set of studies that cannot be well summarized within the limitations of this proposal. Collectively the studies span thousands of students and teachers and hundreds of schools. Results: We found that the variance decomposition results suggested substantial clustering among teacher mediators within schools as well as with students in classrooms and schools (Tables 1-5). The results also suggested that common covariates (e.g. minority status, SES, pretests, etc.) explained variance in the outcomes and mediators. Implications: Consider an example in which schools are randomly assigned to treatment conditions (e.g., professional development program), we observe a classroom-level mediator and record a student-level outcome (ELA achievement). In planning a study, we might inquire as to approximately how many schools we need to have a reasonably high chance of detecting the mediation effect if it exists. Using the models and statistical power formulas in the literature (Kelcey, et al., 2020), let us assume we anticipate the following parameter values for our study: The resulting power curve as a function of the school sample size is plotted in Figure 1. Based on the assumed empirical parameter values, we would expect that sampling about 40 schools would yield about a 0.80 chance of detecting the mediation effect under a three-level school-randomized design. Conclusions: [Please see abstract PDF for full abstract text.]
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A