NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED663431
Record Type: Non-Journal
Publication Date: 2024-Sep-20
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Evidence Factors in Fuzzy Regression Discontinuity Designs with Multiple Control Groups for Evaluating Extended Time Accommodations
Youmi Suk; Youjin Lee
Society for Research on Educational Effectiveness
Background/Context: Some observational studies involve multiple layers of treatment selection, specifically in the context of the extended time accommodation (ETA) for English language learners (ELLs). In ETA settings, the first selection occurs due to the eligibility rule, where students whose ELL English proficiency is below a certain threshold are eligible for ETA. This forms a regression discontinuity (RD) design. The second selection occurs by school administers who determine students' ETA receipt status under school-specific rules and constraints. Then, the third selection is made by students' decision on whether to use the ETA offered (Suk et al., 2022). Each layer of selection is influenced by different selection biases and requires different data analyses. If carefully designed, multiple analyses can allow for constructing "evidence factors." Evidence factors provide two or more independent tests of the same null hypothesis about the treatment effect, each potentially subject to different biases (Rosenbaum, 2010). For example, potential bias in the RD analysis does not affect the validity of other analyses based on other layers of selection. By combining these multiple evidence factors, researchers can strengthen causal conclusions even with a single dataset while separating independent sources of biases among them (Rosenbaum, 2023; Zhao et al., 2022). Purpose/Objective: This study investigates how to combine three evidence factors associated with ETA assignments in fuzzy regression RD designs, where ELL English proficiency serves as the running variable and noncompliance occurs through two mechanisms. Using three different treatment statues in the ETA context, we construct three evidence factors based on one treated and three different control groups. Each evidence factor aims to test the same null hypothesis--no effect of each student making use of ETA--using a stratified nonparametric test. For constructing one evidence factor, we utilize the local randomization framework in RD contexts (Cattaneo et al., 2015) that can be seamlessly applied to the randomization-based inference. We then combine p-values from these three (nearly) independent tests to reinforce causal conclusions on the ETA effect. We also evaluate the effectiveness of our approach via a simulation study and a real dataset from the National Assessment of Educational Progress (NAEP). The NAEP Assessment: Setting: The NAEP assessment used in this study is the largest continuing and nationally representative assessment in the United States (Oranje & Kolstad, 2019). Population: Our study focuses on 8th grade students with the advanced ELL proficiency who do not have disabilities. Program: The ETA program is designed to provide ELL or students with disabilities, up to three times the regular testing time during the NAEP assessment. Research Design: We employ our proposed approach on evidence factors to evaluate the effect of using ETA on student achievement. We use ELL English proficiency categories as the running variable in the RD design. Data Analysis: We analyze data from the 2017 NAEP assessment for grade 8, along with process data that contains approximately 28,000 students. Using process data, we identify students who made use of ETA during the test. Findings/Results: We present how to construct three evidence factors based on three different selection mechanisms in ETA settings. These multiple selection mechanisms form four groups: ETA users (Treatment), non-users (Control 1), non-recipients (Control 2), and ineligibles (Control 3); see Table 1. The first evidence factor comes from a comparison between ETA users (Treatment) versus non-users (Control 1). The second factor is about a comparison between ETA recipients (i.e., Treatment + Control 1) versus nonrecipients (Control 2). The third factor is based on the RD analysis that compares eligible students (i.e., Treatment + Control 1 + Control 2) versus ineligible students (Control 3), which are determined by their running variables. Importantly, the ETA effect can only occur when students make use of it; otherwise, there is no assumed ETA effect. Consequently, each of the three proposed evidence factors supports the null of no ETA effect under each of their sharp null. These three evidence factors are intentionally designed in this way to separate bias from one factor to the others, despite the potential for underpowered analyses in the second and the third factors, which include one or two control groups in the updated "treatment" group. Our posited assignment models for three treatment assignments are as follows: [equations omitted] where K[subscript k] (k =1,2,3) represents an arbitrary function of the measured covariates w[subscript ij] of individual i in stratum j, where the strata are formed using the observed covariates. [gamma][subscript k] represents the impact of the unmeasured covariate u[subscript ij,k]. "F" represents a collection of the potential outcomes and measured and unmeasured covariates, and "W" represents the window around the cutoff where local randomization is valid. Given the assignment models, we conduct a simulation study to evaluate the validity and the performance of our proposed approach. In simulations, we specifically use near-optimal generalized full matchings, so that we can match the treatment group and three different control groups simultaneously on the observed covariates. We then perform Hodges-Lehmann aligned rank tests with strata for each evidence factor and combine p-values from these three nearly independent tests via Fisher's method. We further perform the sensitivity analysis by varying the parameter value of [gamma][subscript k]'s. Lastly, we provide the demonstration of our proposed approach in the 2017 NAEP assessment and process data. Conclusions: This study will strengthen the causal evidence for evaluating the effect of making use of ETA by synthesizing three evidence factors that have been carefully tailored to the ETA context. The proposed evidence factors in fuzzy RD designs with multiple control groups from this study have the potential to be applied not only in other educational contexts but also beyond the field of education.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Junior High Schools; Middle Schools; Secondary Education; Elementary Education; Grade 8
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Assessments and Surveys: National Assessment of Educational Progress
Grant or Contract Numbers: N/A