ERIC Number: ED657183
Record Type: Non-Journal
Publication Date: 2021-Sep-28
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Regression Discontinuity Designs with an Ordinal Running Variable: Evaluating the Effects of Extended Time Accommodations for English Language Learners
Youmi Suk; Peter M. Steiner; Jee-Seon Kim; Hyunseung Kang
Society for Research on Educational Effectiveness
Background/Context: Regression discontinuity (RD) designs are used for policy and program evaluation where subjects' eligibility into a program or policy is determined by whether an assignment variable (i.e., running variable) exceeds a pre-defined cutoff. Under a standard RD design with a continuous assignment variable, the average treatment effect (ATE) at the cutoff can be non-parametrically identified and can be estimated by comparing the average outcomes among subjects "just below" and "just above" the cutoff (Hahn, Todd, & van der Klaauw, 2001; Imbens & Lemieux, 2008; Lee & Lemieux, 2010). However, often in practice, the running variable that determines subjects' eligibility into the program is based on discrete or ordinal variables. For example, to evaluate the effectiveness of extended time accommodations (ETA) on test scores, the National Assessment of Educational Progress (NAEP) assessment used students' ELL English proficiency categories, an ordinal variable with six levels (No Proficiency, ELL Beginning, ELL Intermediate, ELL Advanced, Formerly ELL, and Never ELL) as the eligibility criterion. Specifically, students with ELL Advanced (i.e., the cutoff) or lower level were eligible to receive ETA. Due to the ordinal running variable, the ATE at the cutoff cannot be identified with traditional techniques because the notion of "just below" and "just above" does not exist for discrete variables. Purpose/Objective/Research Question: This paper provides a framework to use RD with an ordinal running variable by adapting the identification and estimation strategy proposed by Lee and Card (2008) to ordinal, discrete variables. The primary strategy maps the ordinal running variable onto a numeric scale based on a scaling function and uses mixed effects regression models to extrapolate near the cutoff. We also present sensitivity analyses to check the conclusions' robustness to (i) the choice of the scaling function, (ii) the choice of the cutoff value, and (iii) an unobserved confounder. We use our approach to assess the effects of ETA on students' math proficiency in the 2017 NAEP study. The NAEP Assessment/Setting/Population/Program/Research Design/Data Analysis: We briefly provide background information on the program (ETA) and the NAEP study. Setting: The NAEP assessment is the largest nationally representative and continuing assessment (Oranje & Kolstad, 2019), and the 2017 NAEP assessment was conducted for grades 4 and 8 in mathematics, reading, and writing. Population: Our target population is 4th grade students whose English proficiency level is at the Advanced ELL category, but do not have disabilities. Program: ETA is available only to ELL or students with disabilities (SD). It provides them with up to 3 times of the regular time, i.e., 90 minutes per block in the NAEP assessment (Kim & Circi, 2018, 2019). In the 2017 NAEP Grade-4 data, about 8.7% of the original reporting sample was ELL only, 11.7% was SD only, and 1.5% had both ELL and SD status. About 10.6% of the sample received ETA. Research Design: We use the aforementioned RD design to evaluate ETA's effects and we use ELL English proficiency categories as an ordinal running variable. Data Collection and Analysis: Our final analysis sample consisted of 116,910 students from 7,450 schools (78.2% of the original sample). Note that numbers are rounded to nearest tens. Our proposed analysis is based on using an RD design with an ordinal running variable. The outline of our identification and estimation plan is summarized above. We also conduct sensitivity analyses to assess the sensitivity of our conclusions to design-related features. Findings/Results: In this section, we formalize the key identifying assumptions for RD designs with an ordinal running variable and we highlight our estimation strategy. Let A[subscript ij] [element-of] {0, 1} be a binary treatment variable for student "i" from school "j." Treatment status is determined by ordinal running variable X[subscript ij] where each category [omega subscript 1],..., [omega subscript K] is assigned to a numerical scaling function S([omega subscript k]) = x[subscript k](k=1,..., K); for our study, A[subscript ij] = 1 if X[subscript ij] [less than or equal to] x[subscript c] and A[subscript ij] = 0 if X[subscript ij] > x[subscript c] where a cutoff score x[subscript c] is the scale value of the Advanced ELL category. Let Z[subscript ij] [element-of] {0, 1} be a binary treatment received variable where Z[subscript ij] = 1 if student "ij" received ETA and otherwise, Z[subscript ij] = 0. Let Y[subscript ij] be the observed outcome. We make the following assumption on the outcome: (A1) Outcome Regression Function: E[Y[subscript ij]|X[subscript ij] = x[subscript k]] = A[subscript ij][tau](x[subscript c]) + h[subscript S](x[subscript k]) with h[subscript S](x[subscript c])=E[Y[subscript ij](0)|X[subscript ij] = x[subscript c]] Here, h[subscript S](x[subscript k]) is a continuous parametric function with respect to the chosen scaling function S([omega subscript k]) = x[subscript k]. The requirement h[subscript S](x[subscript c])=E[Y[subscript ij](0)|X[subscript ij] = x[subscript c]] guarantees that the expected potential control outcome Y[subscript ij](0) at the cutoff score is predicted by h[subscript S](x[subscript c]). The parameter [tau](x[subscript c]) is the ATE (i.e., ITT) at the cutoff x[subscript c]. To identify the LATE, we make the following assumption on the treatment received variable: (A2) Treatment Regression Function: E[Z[subscript ij]|X[subscript ij] = x[subscript k]] = A[subscript ij][alpha] + g[subscript S](x[subscript k]) with g[subscript S](x[subscript c])=E[Z[subscript ij](0)|X[subscript ij] = x[subscript c]] Here, g[subscript S](x[subscript k]) is a continuous parametric function with respect to the scaling function S([omega subscript k]) = x[subscript k]. The term [alpha] represents the discontinuity in treatment probabilities at the cutoff and is used to estimate LATE at the cutoff. Under the assumptions above, we estimated the ITT and LATE of ETA on math scores at the cutoff. We found that among ETA-eligible students (i.e., ELL), about 32.5% actually received ETA. We also found that there is no evidence of an ITT (i.e., the effect of ETA eligibility) on math scores at the cutoff, whereas there is strong evidence of a LATE (i.e., the effect of receiving ETA among complier students) on math scores at the cutoff. Also, our sensitivity analyses revealed that our ITT estimate was sensitive to the choice of the cutoff value. But the LATE estimate was insensitive to the choices of the scaling function and the cutoff value, and an unmeasured confounder. Conclusions: In this paper, we proposed to use an RD design with an ordinal running variable. Despite the additional assumptions necessary to identify the ATE (i.e., ITT) and the LATE, our proposed approach yields insights about the effectiveness of ETA on test scores and can be a useful tool to evaluate the effects of testing accommodations in educational studies.
Descriptors: Educational Policy, Eligibility, Cutting Scores, Testing Accommodations, Scores, English Language Learners, National Competency Tests, Language Proficiency, Correlation, Evaluation Criteria, Identification, Robustness (Statistics), Scaling, Mathematics Achievement, Grade 4, Grade 8, Mathematics Tests, Reading Tests, Writing Tests, Students with Disabilities, Regression (Statistics), Policy Analysis
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Education; Grade 4; Intermediate Grades; Grade 8; Junior High Schools; Middle Schools; Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Assessments and Surveys: National Assessment of Educational Progress
Grant or Contract Numbers: N/A