ERIC Number: ED599855
Record Type: Non-Journal
Publication Date: 2017-Apr-28
Pages: 10
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-
EISSN: N/A
Waiting for Baselines to Stabilize: Consequences of Response-Guided Experimentation on Meta-Analyses of Single-Case Studies
Ferron, John M.; Joo, Seanghwane
AERA Online Paper Repository, Paper presented at the Annual Meeting of the American Educational Research Association (San Antonio, TX, Apr 27-May 1, 2017)
Single-case researchers frequently adopt a form of response-guided experimentation where decisions about the design of the study are made based on an ongoing visual analysis. For example, multiple-baseline researchers may delay the start of intervention until data document a stable baseline pattern so that baseline trends can be reliably extended, or the researchers may wait for a case to respond to intervention prior to intervening with the next case. Although well intentioned, these response-guided experimental strategies may substantially bias the intervention effect estimates. To illustrate, consider a study where the intervention was designed to increase a behavior that in the absence of the intervention would not trend up or down. Although the true baseline slope parameter is zero, it is unlikely that the slope of the observed values would be exactly zero. If by chance the slope happened to be negative after gathering the minimum number of baseline observations the researcher would intervene, but if the slope were notably positive the researcher would gather another observation, or two, or three, each time looking for a flat or downward trend. When the more acceptable baseline emerged, the researcher would intervene. If this study were to be replicated many times, the average baseline slope would be negative, not zero, and as a consequence the intervention effect estimate would be positively biased. Meta-analyses of single-case studies are likely to include studies that used response-guided experimentation and thus it is important to understand the degree to which these practices may bias treatment effect estimates. If the biases are minimal then it may be reasonable to ignore the response-guided nature of the data collection in the meta-analysis of single-case studies. If the biases are substantial, however, then it is important to determine if there are ways of responding to the data that meet the needs of the researchers but create fewer problems for effect estimation and meta-analyses. The purpose of this Monte Carlo simulation study is to evaluate which response-guided practices and which estimation approaches (within- versus between-series estimators) provide the least biased effect estimates. An algorithm was written to simulate a variety of response-guided practices, including extending phases when there is baseline instability, operationalized as non-negligible trend in the direction of expected change or non-negligible trend in either direction, as well as practices that look for a response to intervention prior to intervention with each successive case. Design factors for the simulation study include the response-guided practices, as well as the number of cases per study, the number of studies included in the meta-analysis, and the parameter values of the generating model. Both within- and between-series estimates of the treatment effect are obtained for each data condition using multilevel meta-analytic models. The bias and RMSE for each type of treatment effect is estimated for each data condition based on 2000 simulated meta-analyses. Based on preliminary runs of the pilot code the simulation will be completed in September.
Descriptors: Research Design, Bias, Meta Analysis, Computation, Response to Intervention, Outcomes of Treatment, Research Problems
AERA Online Paper Repository. Available from: American Educational Research Association. 1430 K Street NW Suite 1200, Washington, DC 20005. Tel: 202-238-3200; Fax: 202-238-3250; e-mail: subscriptions@aera.net; Web site: http://www.aera.net
Publication Type: Speeches/Meeting Papers; Reports - Descriptive
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A