NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED658628
Record Type: Non-Journal
Publication Date: 2022-Sep-23
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Intraclass Correlations for Student Science Outcomes Including Teachers as a Level of Nesting
Zuchao Shen; Chris Curran; Walter Leite; Joni Splett; Kyle Cox; Ben Kelcey
Society for Research on Educational Effectiveness
Purpose: When designing cluster-randomized trials, researchers must consider the efficient use of resources to determine the research design with adequate statistical power (i.e., the probability of detecting treatment effects if they exist). The validity of power analysis results depends on accurate inputs or design parameters, such as intraclass correlations (ICCs) that quantify the correlations among individuals in the same clusters (Hedges & Hedberg, 2007, 2013; Konstantopoulos, 2008a, 2008b). This study aims to use a nationally representative sample and compile empirical ICCs for science outcomes in settings of students nested within teachers nested within schools. The results of this study can help design efficient and effective three-level evaluations (cluster-randomized trials or multisite-randomized trials) estimating the effects of interventions involving teachers on student science outcomes. Related Literature: The literature has compiled empirical ICCs across a wide range of outcome domains, grades, and continental regions (e.g., Bloom et al., 2007; Hedges & Hedberg, 2007, 2013; Kelcey et al., 2016; Kelcey & Phelps, 2013; Raudenbush et al., 2007; Westine et al., 2013). However, most of these studies focused on designs with students nested within schools (e.g., Bloom et al., 2007; Hedges & Hedberg, 2007; Kelcey et al., 2016) or students nested within schools nested within districts (e.g., Hedges & Hedberg, 2013; Westine et al., 2013). The omission of a teacher level of nesting makes these studies provide little guidance on designing the evaluation of programs involving teachers (e.g., teacher professional development and innovative curriculum initiatives; Hill et al., 2020; Jennings et al., 2017; Kelcey et al., 2019; Lynch et al., 2019; Nye et al., 2004). Omitting a teacher level of nesting in design and analysis may lead to severe consequences, such as inflated type I error rates and spurious results (e.g., Moerbeek, 2004; Opdenakker & Van Damme, 2000; Van den Noortgate et al., 2005). Only a few studies compiled empirical ICCs with students, teachers, and schools as the levels of nesting (Dong et al., 2016; Jacob et al., 2010; Zhu et al., 2012). Still, these studies covered a limited range of grades and outcome domains and were primarily estimated from data collected in experimental studies. Data and Measures: This study uses the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K), a nationally representative sample of kindergarten children during 1998-99. ECLS-K study followed students from their kindergarten year through eighth grade, and it collected information in the fall and spring of kindergarten (1998-99), the fall and spring of first grade (1999-2000), the spring of third grade (2002), the spring of fifth grade (2004), and the spring of eighth grade (2007; Tourangeau et al., 2006). The kindergarten and first grade assessed student's general knowledge, including basic science concepts and concepts in social studies. These assessments measured conceptual understanding of scientific facts, skills and abilities to form questions about the world, to try to answer them based on tools and the evidence, and to communicate answer and how the answers were obtained. Social studies materials included questions relating to history, culture, geography, and economics (Tourangeau et al., 2006). ECLS-K science and measurement was developed through consultation with experts in child development, elementary education, and content experts (Tourangeau et al., 2006). This study focuses on science outcome measures near the end of each academic year (i.e., spring 1999, spring 2000, spring 2002, spring 2004, and spring 2006). It is common to evaluate students' performance near the end of an academic year. Consistent with previous studies (e.g., Jacob et al., 2010), we use IRT scores for the analysis. The sample sizes used in the analysis vary across grade levels. For example, the average sample sizes are 13,202 students with 4,160 teachers in 2,102 schools for all schools sample, and they are 5,900 students with 2,152 teachers in 1,051 schools for the low achievement school sample (Table 1). Design Dimensions: Our report focuses on ICCs for study designs involving teachers as a nesting level with random assignments at the teacher or school level. We considered several design dimensions similar to previous reports (Hedges & Hedberg, 2007). Design dimensions include students' grade level, the set of covariates used in the study design and analysis, and the achievement status of schools among the overall population (e.g., all schools and low achievement schools with mean scores below the average). Analytical Models: We use the R package lme4 (Bates et al., 2015) to conduct data analyses with the restricted maximum likelihood method, providing a more accurate estimation of the variance components than the maximum likelihood estimation. Our analysis does not use design weights, similar to the previous literature (e.g., Hedges & Hedberg, 2007). Doing so, we can compute the 95% confidence intervals for unconditional ICCs through the bootstrap parameter estimation method in the lme4 package. Specific models are not presented due to space limitations. Results: For all schools, the ICCs at the school level are larger than those at the teacher level (Table 1). For low achievement schools, the ICCs at the school level are much smaller (Table 1). The ICC at the teacher level increases along with aging or schooling (Tables 1). The inclusion of pretest scores can help explain a considerable portion of outcome variance at different levels (Tables 2). Including demographic covariates provides little extra explanatory power to explain the variances at any level when the pretest scores are already included (Tables 2). As illustrated in Table 3, the results of this study can help design efficient and effective three-level designs under a recently developed optimal design framework that considers sampling costs (2 citations masked). Conclusion: Identifying efficient and effective sampling plans depends on both the development of design theories and compiling of empirical design parameters that mirror practices in the field. This study compiles the ICCs for science outcomes to allow interventions to be improved through crucial stakeholders (teachers) and illustrates the utility of these parameters in a recently developed optimal design framework. More broadly, this study demonstrates the importance of reporting and using grade-specific ICCs especially involving teachers and science outcomes.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A