NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED658629
Record Type: Non-Journal
Publication Date: 2022-Sep-22
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Closing the Gap in Student Achievement: Application of the Augmented Errors-in-Variables Method
Robert Meyer; Sara Hu; Michael Christian
Society for Research on Educational Effectiveness
This paper develops models to measure growth in student achievement with a focus on the possibility of differential growth in achievement for low and high-achieving students. We consider a gap-closing model that evaluates the degree to which students in a target group -- students in the bottom quartile of measured achievement -- perform better than expected, given a standard model of student achievement. Since student achievement is inevitably measured with error, the models implement methods that control for measurement error. The distinctive aspect of model is that true prior achievement and the achievement quartile indicator are tightly connected yet need to be treated differently since the model specification requires measurement error control be applied to measured prior achievement, but not to the bottom quartile indicator which is based on measured prior achievement. We demonstrate how the standard errors-in-variables (EV) method does not address this need and develop a new method that does address this need: augmented errors in variables (AEV). We show that other methods of controlling for measurement error such as SIMEX also do not address this need. The gap-closing model was developed in response to a school report card redesign initiative recently launched in a state. This initiative expanded the state school report card to focus on the growth performance of students in the bottom quartile of prior achievement. In the Fall of each year students in each school are identified as being in the gap closing target group if they scored in the bottom quartile in on the previous state assessment. The names of these students are provided to each school with the expectation that schools will implement policies and practices to improve the achievement of these students, as measured by the end-of-year state assessment. The state publishes in the subsequent school report card data on the growth in student achievement in both the target and non-target groups. The new gap-closing model, and associated estimation method, is designed to measure the differential performance of schools and the state overall with respect to achievement growth of the target (bottom quartile) group and non-target (upper quartile) group. The need for an augmented approach to controlling for measurement error arises from the fact that it is necessary to simultaneously treat the bottom quartile indicator as not measured with error (exogenous) and the continuous pretest variable as measured with error (endogenous), since these two variables are inextricably connected. The AEV method addresses this problem in two steps. First, a measure of true prior achievement is constructed, given measured prior achievement and the other control variables included in the model, including demographic variables and school effects. The constructed measure replaces the fallible pretest variable. Second, the vector of predictor variables, including the fallible pretest variable, are used as instrumental variables (IV). This approach necessarily reverses the traditional IV method where a set of instrumental variables is used to "instrument" variables measured with error. An indirect least squares ILS) interpretation of the AEV/IV method perhaps best illustrates why the new method works. The two reduced form equations in the ILS system include: (1) a regression of the dependent variable on the vector of measured variables (the instrumental variables) and (2) a constructed regression of the vector of variables measured without error on the instrumental variables. Since the model is exactly identified, the structural parameters from the AEV/IV and AEV/ILS approach are identical. One useful property of the AEV method is that it yields estimates that are identical to the EV method when in a given model application the latter estimation method is appropriate. As a result, the data set constructed to allow application of the AEV method can also be used to estimate other, possibly, restricted models. We have applied the AEV method to Monte Carlo data to study the properties of the model and to student data obtained from state that where the gap-closing initiative has been implemented. The Monte Carlo analyses indicate that the AEV method yields consistent parameter estimates and that estimates of standard errors obtained from the standard IV software are only slightly smaller than those obtained from the Monte Carlo replications. More accurate estimates can be obtained from application of the bootstrap. We have estimated the gap-closing model using pre-COVID data for posttest years 2018 and 2019. We will report estimates at the conference using data for the 2021-2022 school year, which will be available this summer. Although we are interested in the bottom quartile effects -- the overall average effect and separate effects for each school -- we are even more interested in how these effects have changed over time, before and after implementation of the gap-closing initiative. Our interest in the change in the gap-closing parameter is driven in part by two concerns. One, the gap-closing coefficient estimated prior to the initiative may be positive because of programs and policies targeted at low achievers that existed prior to the initiative. Two, the true relationship between post and prior achievement is not guaranteed to be linear. As in a before and after evaluation design, the change in the gap-closing parameter is a better measure of the impact of the initiative then the estimate based on a single year. Estimates of the gap-closing parameter based on 2018 and 2019 data indicate that, as expected, the EV estimate is severely biased upward, on the order of a 0.30 effect size. The AEV estimate is positive but much smaller, about 0.05, depending on the grade and subject. As indicated above, we will report at the conference whether the gap-closing initiative has increased the bottom quartile effect compared to the estimates obtained prior to the pandemic. Although our focus in this paper is on using the AEV method to estimate the gap-closing model described above, we also consider how the method could be applied to other complex models where variables are measured with error. One such model is a spline model where the spline variables are based on variables measured with error.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A