ERIC Number: ED659420
Record Type: Non-Journal
Publication Date: 2023-Sep-29
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
The Impact of CSI Designation on Student Outcomes
Drew Atchison; Umut Ozek; Kerstin Le Floch; Damon Blair; Steve Hurlburt
Society for Research on Educational Effectiveness
Background/significance: Standards-based accountability systems have been a pillar of education reform in the United States for almost three decades. Although accountability systems have evolved in progressive reauthorizations of the Elementary and Secondary Education Act (ESEA), the underlying theory of action has changed little. The basic premise is that school-level accountability designations signal low (and high) levels of student performance or school quality, provide meaningful information to local stakeholders about actionable areas in which schools are underperforming, and motivate educators to foster improvements in student outcomes (Elmore, 2004; Hanushek & Raymond, 2001; National Research Council, 2011; O'Day, 2002; Saw et al., 2017). In December 2015, ESEA was reauthorized as the Every Student Succeeds Act (ESSA), which made several key changes to federal accountability policy. A key feature of ESSA is its flexibility, as it gave states substantial latitude in designing their processes for identifying and providing supports to low-performing schools (i.e., comprehensive support and improvement [CSI] schools). The current policy context promotes increasingly complex accountability systems with multiple measures and varied ways of calculating school performance, while also providing states, districts, and schools with more flexibility regarding how they will address low performance. Increased complexity and flexibility mean that school leaders must weigh more information and make strategic decisions about which accountability measures to prioritize and what improvement strategies to use in response to accountability results. Although several recent meta-analyses suggest that on average school performance improves in schools identified as low-performing in ESSA-style accountability systems, there is substantial heterogeneity in results with some systems being successful and others potentially harming low-performing schools (Redding & Nguyen, 2020; Schueler et al., 2020). Despite 25 years of federal accountability policies, many questions still exist about their ability to spur broad improvement, and notable gaps exist in our understanding of how accountability systems work, particularly in the context of the ESSA reauthorization. Purpose/Research Questions: In this study, we examine the causal effects of CSI designation on student outcomes including test scores, suspensions, absenteeism, high school graduation, and other student outcomes that are used for school accountability purposes. We also examine some of the mechanisms that might be driving the observed effects such as student mobility and teacher attrition. Setting: We have partnered with three states to conduct analyses of CSI schools: California, Ohio, and Florida. These three states represent different approaches to accountability and three different state contexts. In California, CSI schools are identified based on a series of business rules, where each accountability "indicator" is assigned a color (red being the lowest performing and blue being the highest performing). Schools with a sufficient number or percentage of red and orange indicators are identified as CSI. Both Ohio and Florida use index-based approaches for identifying CSI schools, where a variety of accountability measures are aggregated based on weights assigned to each measure. Schools with a low enough "index" are identified as CSI. Population/Sample: The population consists of all schools in our three partner states. Specifically, as of 2019-20, California had approximately 750 CSI schools, Florida had approximately 440 CSI schools, and Ohio had approximately 230 CSI schools. Intervention: Under ESSA, each state is required to identify at least the bottom 5% of schools based on performance and high schools with graduation rates below 67% to be identified as CSI schools. These schools are then intended to receive some additional funding, be provided additional supports, develop plans for improvement, implement strategies to improve, and ultimately improve student outcomes. Under ESSA, states were provided more flexibility to determine how to intervene in CSI schools and which improvement strategies to emphasize. Although not included in this paper, we conducted a principal survey and district interviews to better understand approaches to improvement and supports provided to CSI schools. Data sources and methods: To investigate the effects of the CSI designation on student outcomes, we obtained student- and school-level data from three states. Outcome data in each state consist of student-level test scores in math and ELA, attendance rates, student discipline, and graduation. The three states also provided student-level demographic data including student race, economic disadvantage status, special education status, and English learner status. We paired student-level data with data on CSI designation in 2018-19 and subsequent years. Using this data, we used two approaches to examine the effects of CSI designation on student outcomes. First, we used regression discontinuity (RD) to investigate the effect of CSI designation around school performance cutoffs. Because both Ohio and Florida both use an aggregate school performance index where schools performing below a cutoff are designated as CSI schools, the RD approach is most straightforward in these states. In California, a series of cutoffs on each individual indicator are used to determine performance ratings on each indicator. Given that no single cutoff determines CSI status, we used a comparative interrupted time series approach for California that establishes performance trends in both CSI and non-CSI schools prior to initial designation and then compares the difference in trends for both school types in the post-identification time periods. However, we have also used RD in California to understand the effect of earning a red or orange color on a given indicator. Findings: Preliminary results from Ohio through four years after CSI designation (2018-19 through 2021-22) using RD show no statistically significant improvements in student outcomes for CSI schools relative to non-CSI schools and several areas were CSI schools performed worse than non-CSI schools, including graduation rates, attendance, and ACT test scores. We are working on developing potential explanations for these findings. We are also working on completing analyses for California and Florida.
Descriptors: Educational Improvement, Outcomes of Education, Accountability, Educational Quality, Academic Achievement, School Effectiveness, Educational Legislation, Elementary Secondary Education, Federal Legislation, Educational Indicators
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Ohio; Florida; California
Identifiers - Laws, Policies, & Programs: Every Student Succeeds Act 2015
Grant or Contract Numbers: N/A