ERIC Number: ED659509
Record Type: Non-Journal
Publication Date: 2023-Sep-29
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Innovation in Evaluation and Accountability Systems Aligned with Diversity, Equity, and Inclusion
Robert Meyer; Tracy Diel; Rinor Jahiu; Hayley Tymeson
Society for Research on Educational Effectiveness
Background: This paper considers a new policy and statistical framework for evaluating K-12 schools and policies that aligns with diversity, equity, and inclusion values. The new approach broadens the standard approach to accountability and evaluation by combining features of evaluation and multi-level growth models with approaches used in systems of early warning, college/career/workforce readiness, and predictive analytics. A hallmark of these latter systems is that they focus on student outcomes beyond student test scores, including student attendance and suspensions, enrollment in challenging courses, grades, and high school and college graduation. We expand the focus to outcomes such as participation in English language learner (ELL) and other programs. The expanded focus on a broader set of student outcomes supports a DEI focus by providing a more complete portrait of the malleable factors that affect students with very different demographic backgrounds. An important feature of the new framework is that by focusing on post high school college and career outcomes, the framework connects K-12 outcomes to the long-term outcomes that students, parents, and educators value. Research Questions: (RQ1) How is it possible to connect K-12 outcomes at the elementary, middle, and high school grades with medium and long-term outcomes that range from reading proficiency in 3rd grade to post high school outcomes such as college graduation? (RQ2) How is it possible to incorporate extensive K-12 data in each grade into models of high school and post high school outcomes to promote statistical precision, transparency, and feasibility? (RQ3) What are the appropriate statistical tools to highlight the DEI dimensions of K-12 student and school-level outcomes? Setting and Participants: The paper addresses key policy and statistical issues and progress in developing and implementing a DEI-aligned evaluation and accountability system in collaboration with four large, but diverse, school districts. Two of the districts are especially interested in extending these systems to incorporate Pre-K-3 data, which has less commonly been included in these systems, given the bias toward federally required assessments in grades 3-8. The other two districts are especially interested connecting middle and high school outcomes and experiences to post high school outcomes. Research Design: A key objective of the research is to calibrate model parameters that connect K-12 outcomes with long-term outcomes. These calibrated coefficients can then be used to evaluate college and career readiness of middle and high school students while the students are still enrolled in school, rather than waiting until these long-term outcomes are observed. To address RQ1, it is necessary to connect K-12 outcomes with long-term outcomes, time spans that in some cases are exceptionally long. The simplest strategy is to use long-span longitudinal panels to connect, for example, 8th grade outcomes with college graduation, a period that covers 11 years of data. This is generally not the preferred approach, however, because longitudinal data is often not available for that length of time and calibrated parameter estimates based on long-span data reflect school policies and contextual factors (such as the COVID pandemic) that likely fail to capture the up-to-date relationships between the outcomes. Our preferred approach is to estimate separate short and medium-span models and chain link them together to produce calibrated coefficients that reflect more up-to-date relationships between the variables. Our research addresses the econometric conditions that must be satisfied to validly implement this approach. A key condition is that the short and medium-span models must be connected using very strong variables. The connecting variables must be strong outcomes in the first model and strong predictors in the second model. We have demonstrated that data measured in the first year after nominal high school graduation satisfies this requirement. To address RQ2, we consider options for creating variables that efficiently and transparently incorporate: (a) potentially large amounts of information and (b) information that may not be available for all students. The first task is especially necessary for models based on high school transcript data where, in principle, student differences in measures and outcomes in all four grades could be salient. We consider alternative data reduction approaches to create parsimonious measures that focus on four major dimensions: attendance and suspensions, course taking, test scores, and grades. To address the second task, we employ multiple imputation to address missing data. This approach is especially valuable in K-3 models where different schools and types of students vary in their use and coverage of different data elements. To address RQ3, we expand the standard multilevel modelling approach to estimate nonlinear models that allow for flexible modelling of within and between school effects of different demographic variables. This approach allows us to assess the degree to which outcomes such as enrollment in Algebra differ due to between-school differences in poverty, within-school differences in poverty, or both. We find, for example, that differences in Algebra enrollment in one of our districts is strongly related to both student poverty and school-level poverty. This framework allows for a more detailed and nuanced measures of the efficacy of schools and policies. Data sources: The study uses student and school-level data on all students from all four districts included in the research. As discussed above, the study utilizes an extensive battery of data, including K-12 and post high school data. The latter data were obtained from the National Student Clearinghouse. In all districts, data is available from school years 2016-17 to 2022-23 (the latter available this Summer) and thus reflects student growth during three periods: pre-COVID, peak-COVID, and "post" COVID recovery. Findings and Conclusions: We have developed and demonstrated options for building a new policy and statistical framework for evaluating K-12 schools and policies that aligns with diversity, equity, and inclusion (DEI) values. To do so, we have addressed the three research questions considered above. The new framework incorporates a much richer set of variables than typically considered in evaluations and accountability systems and provides tools for considering the DEI dimensions of these systems.
Descriptors: Accountability, Inclusion, Diversity, Equal Education, Kindergarten, Elementary Secondary Education, English Language Learners, Career Readiness, College Readiness, Outcomes of Education, Statistical Analysis, School Districts, Measurement Techniques, Evaluation Methods, Longitudinal Studies, Middle School Students, High School Students, School Policy, Correlation, Data Analysis, Institutional Characteristics, Poverty, Algebra, Enrollment Trends, School Effectiveness, Academic Achievement, COVID-19, Pandemics
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Early Childhood Education; Elementary Education; Kindergarten; Primary Education; Elementary Secondary Education; Junior High Schools; Middle Schools; Secondary Education; High Schools
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A