ERIC Number: ED659468
Record Type: Non-Journal
Publication Date: 2023-Sep-29
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Fuzzy Regression Discontinuity Designs with Multiple Control Groups for Evaluating Extended Time Accommodations
Youmi Suk; Yongnam Kim
Society for Research on Educational Effectiveness
Background/Context: Observational studies often employ regression discontinuity (RD) designs and multiple control-group designs to explore the causal quantities of interest. RD designs assess policy and program effectiveness by assigning subjects to treatment based on whether they exceed a pre-defined cutoff. RD designs are classified into two types depending on subjects' compliance with the assigned treatment status. In particular, when compliance is not complete, fuzzy RD designs are used to estimate the local average treatment effect (LATE) at the cutoff (Imbens & Lemieux, 2008). Multiple control groups arise when there are various reasons for students' non-participation in treatment, and their value depends on the availability of supplementary information about unobserved biases. For example, when one control group produces positive unobserved bias and the other produces negative bias, the two control groups can yield consistent and unbiased estimates of bounds on the treatment effect (Rosenbaum, 1987). However, despite the potential benefits of using multiple control groups, there has been limited research on their utilization in RD designs. Purpose/Objective: This paper aims to combine fuzzy RD designs and multiple control-group designs to evaluate the effect of extended time accommodations (ETA) on student performance. Two control groups are formed in ETA settings, one for students who did not receive ETA and another for students who received ETA but did not use it. Our approach enables causal identification and estimation of the average treatment effect on the treated (ATT) at the cutoff from the fuzzy RD design, along with its bounds formed by utilizing the two control groups, which serves as a sensitivity analysis. We also propose flexible estimators based on machine learning (ML) to construct the bounds on the ATT at the cutoff. The NAEP Assessment/Setting/Population/Program/Research Design/Data Analysis: Setting: The NAEP assessment used in this paper is the largest continuing and nationally representative assessment in the United States (Oranje & Kolstad, 2019). Population: Our study focuses on 8-th grade students with the advanced English language learner (ELL) proficiency who do not have disabilities. Program: The ETA program is designed to provide ELL or students with disabilities, up to three times the regular testing time during the NAEP assessment. Research Design: We employ our proposed RD design with multiple control groups to evaluate the effect of ETA on student achievement. We use ELL English proficiency categories as the running variable in the RD design. Data Analysis: We analyze data from the 2017 NAEP assessment for grade 8, along with process data containing approximately 28,000 students. Using process data, we identify students who made use of ETA during the test. Findings/Results: We present the key identifying assumptions for a fuzzy RD design and outline our estimation strategy for constructing causal bounds using multiple control groups. We define a binary treatment variable A[subscript ij] [element of] {0, 1} for student "I" from school "j," which is determined by a discrete running variable X[subscript ij]; specifically, A[subscript ij] = 1 if X[subscript ij] [less than or equal to] x[subscript c] and A[subscript ij] = 0 if X[subscript ij] > x[subscript c] where x[subscript c] is a cutoff score. We also define T[subscript ij] = 1 if student "ij" used ETA and T[subscript ij] = 0 otherwise, and define the observed outcome Y[subscript ij]. In addition to the four identifying assumptions for a fuzzy RD design with a discrete running variable (i.e., outcome regression function, treatment regression function, local monotonicity, and local exclusion restriction) (Lee & Card, 2008; Suk et al., 2022), we require another identifying assumption as: (A5) Local Absence of Always-Takers: Pr(T[subscript ij](1) = T[subscript ij](0) = 1 | X[subscript ij] = x[subscript c]) = 0. Assumption (A5) ensures that the LATE at the cutoff is equivalent to the ATT at the cutoff (denoted as [tau subscript ATT](x[subscript c])) because all the treated cases are compliers, rather than a mixture of compliers and always-takers. This indicates one-sided noncompliance. In our ETA context, students ineligible for ETA are highly unlikely to use ETA, and it renders the existence of always-takers impossible in our setting. To construct the bounds on the ATT at the cutoff, we employ two control groups in the ETA setting. We use T[subscript ij] [element of]{C[subscript 0], C[subscript 1], C[subscript 2]} to indicate the combined ETA receipt and use statuses where ETA users are denoted as C[subscript 0], ETA nonusers (though offered) denoted as C[subscript 1], and ETA non-receivers denoted as C[subscript 2]. We assume that in our ETA context, the first control group (C[subscript 1]) yields positive unobserved bias, whereas the second control group (C[subscript 2]) yields negative unobserved bias; see Figure 1 for a graphical representation of multiple control groups in the ETA setting. Using this information, we derive lower and upper bounds on the ATT at the cutoff as: [equation omitted]. Moreover, we propose utilizing flexible ML-based estimators to construct bounds on [tau subscript ATT](x[subscript c]) to prevent bias from model misspecification. To achieve this, we employ "off-the-shelf" ML-based causal inference methods, notably Bayesian additive regression trees (Hill, 2011), the targeted maximum likelihood estimator (van der Laan & Rubin, 2006) with ensemble learning, and causal forest (Wager & Athey, 2018), to obtain the conditional average treatment effects on the treated. With our causal assumptions in place, we estimate the ATT of ETA on math scores at the cutoff using the two-stage least square estimator and its bounds using the ML-based bound estimators based on the 2017 NAEP assessment and process data. Conclusions: This paper proposed a novel approach that utilizes a fuzzy RD design with multiple control groups to evaluate the ATT at the cutoff along with its bounds. In particular, the use of multiple control groups, identified through process data, enables the construction of the bounds that provide a useful sensitivity check. The proposed approach offers theoretical and practical insights into evaluating the effects of testing accommodations in education research.
Descriptors: Regression (Statistics), Research Design, Control Groups, Program Effectiveness, Testing Accommodations, National Competency Tests, Middle School Students, Grade 8, English Language Learners
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Junior High Schools; Middle Schools; Secondary Education; Elementary Education; Grade 8
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Assessments and Surveys: National Assessment of Educational Progress
Grant or Contract Numbers: N/A