ERIC Number: ED657195
Record Type: Non-Journal
Publication Date: 2021-Sep-27
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Sequential Specification Tests to Choose a Model: A Change-Point Approach
Adam Sales
Society for Research on Educational Effectiveness
Education researchers frequently have to choose between statistical models for their data, and in many cases the candidate models or parameters can be listed in a sequence, m=1,...,M, from less preferable choices to more. For instance, in choosing a bandwidth for regression discontinuity designs, researchers would favor the largest possible bandwidth; in setting criteria for propensity matching, researchers would prefer the set of criteria that includes the largest number of subjects. One option for model specification in these scenarios relies on p-values for goodness-of-fit or specification tests, say p[subscript 1,]....,p[subscript M]. Given a threshold [alpha], often [alpha]=0.05, a researcher could choose the most preferable model that passes the test, "max"{m: p[subscript m]>[alpha]} or the largest model before the first rejection "max"{m*: p[subscript m]>[alpha] [turned A] m[less than or equal to]m*}. However, it is typically unclear how to choose [alpha], or whether the framework of null-hypothesis testing is appropriate for model selection, in which the goal is, essentially, to accept the null hypothesis. This talk will introduce an alternative model selector based on p-values p[subscript 1,]....,p[subscript M] that uses p-values for estimation, rather than for testing, and that does not require researchers to choose a threshold. The method, which originated in the change-point literature (Mallik et al. 2011), proceeds from the observation that when the null hypothesis is true, p-values are distributed as "Uniform"(0,1), with a mean of E[p]=0.5, and when the alternative hypothesis is true, p[right arrow]0 as sample sizes increase. This suggests a least-squares estimate of the optimal model, m[superscript cp]= argmin[subscript m*] [sigma subscript m(less than or equal to)m*](p[subscript m]-1/2)[superscript 2]+[sigma subscript m>m*](pm)[superscript 2], that is, the model choice such that p-values for acceptable models, those with m[less than or equal to]m*, are clustered around 1/2, while those for unacceptable models are small. This change-point selector m[superscript cp] uses the properties of p-values from goodness-of-fit tests to "estimate" the optimal model, rather than to choose which null hypotheses to reject and which to (implicitly) accept. Because m[superscript cp] is based on the full array of p-values, one errant p-value will not drive the entire choice. This talk will evaluate the performance of model selector m[superscript cp], relative to more conventional selection rules, in a simulation study. The simulation study shows that m[superscript cp] represents a compromise between more liberal and more conservative conventional model selectors, and that its performance, while not always the best, is the most consistent across different circumstances of the model selectors included in the study. The talk will also illustrate m[superscript cp] in bandwidth selection for the regression discontinuity design introduced in Lindo, et al. (2010) estimating the effect of academic probation status on college students' subsequent grade point averages (see the attached figure). At each possible bandwidth, we conduct covariate placebo tests, by estimating effects of academic probation on baseline student covariates, and use m[superscript cp], along with threshold-based selectors to choose an optimal bandwidth. We also compare the results to the bandwidth selector proposed by Imbens and Kalyanaraman (2011).
Descriptors: Educational Research, Statistical Analysis, Research Design, Decision Making, Goodness of Fit, Alternative Assessment, Hypothesis Testing, Models, Regression (Statistics)
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A