ERIC Number: ED659661
Record Type: Non-Journal
Publication Date: 2023-Sep-29
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Missing Data Reporting and Handling in Special Education Intervention Research
Lexi Swanz; Allyson Hanson; Daniel R. Espinas
Society for Research on Educational Effectiveness
Introduction: Missing data are bound to occur in education intervention research. Reasons vary but always have the consequence of reducing sample sizes and can, under certain conditions, seriously bias estimated intervention effects. A wide array of methods have been developed for handling missing data (Enders, 2023). Whereas older approaches largely involve simply ignoring missing cases (e.g,. complete case analysis), with newer approaches, missing data can actually be recovered (e.g., multiple imputation; Enders, 2023; Schafer & Graham, 2002). As with any statistical method, though, certain assumptions must be met. Indeed, the appropriateness of any handling method depends on the pattern of missingness and the mechanism thought to underlie it (e.g., missing completely at random; Enders, 2023; Little & Rubin, 2020). Accordingly, information about the pattern, mechanism, and handling method should be clearly reported (Toste et al., 2023). Although missing data reporting and handling have not traditionally been emphasized within special education intervention research (Gersten et al., 2005), recent quality indicator standards have placed an increased focus on these matters (Toste et al., 2023). Though clear recommendations have emerged (Enders et al., 2006), it is unclear what practices are currently used. Previous reviews of the broader education literature indicate that reporting practices are minimal and that researchers largely rely on traditional handling approaches (e.g., complete case analysis; Cheema, 2014). As in other fields, this may reflect gaps in methodological training (Murtonen & Lethtien, 2003). To our knowledge, no review has focused specifically on special education researchers. In this study, we therefore aimed to take stock of missing data reporting and handling practices in special education group design research. Purpose and Research Questions: Our purpose for this review was to summarize missing data reporting and handling practices in special education group design intervention research. We had three primary questions. First, what are the locations (e.g., item, case), amount, and reasons (e.g., attrition) for missing data? Although design and reporting standards in special education specify that authors should clearly describe this information (Gersten et al., 2005), based on previous reviews, we hypothesized that reporting would on average be minimal but vary considerably across studies. Second, what missing data mechanisms (e.g., missing completely at random) are reported and how are they tested? We again hypothesized that reporting would be minimal but vary widely across studies. Third, how are missing data handled? What methods are used and are sensitivity analyses performed? Based on previous reviews, we hypothesized that traditional handling methods (e.g., complete case analysis, pairwise deletion) would be used more often than modern methods (e.g., multiple imputation). Methods: Research Design: Our study is a systematic review. We followed the What Works Clearinghouse process for systematic reviews, which involves defining the review protocol, systematically identifying relevant literature, screening studies for eligibility, reviewing studies deemed eligible, and reporting those findings (Institute of Education Sciences, 2020). Screening and Inclusion Criteria: Our sample of studies was drawn from a larger project aimed at examining how null effects are reported in special education intervention research (Cook et al., 2023). To identify studies for the project, special education journals reported in the 2019 Journal Citation Reports were electronically searched. Studies were included if they (a) used a quantitative group design (randomized controlled trial, quasi-experimental designs with a comparison group without random assignment to groups, regression discontinuity designs) and within-group designs (e.g., single-group, pre/post-test design), (b) empirically evaluated the effect of an intervention on at least one dependent variable, and (c) used inferential statistics to evaluate the effect of the intervention for one or more groups of participants. Studies were excluded if they (a) used a research design other than the specified group designs (e.g., single-case design studies, purely descriptive studies, qualitative studies, secondary data analyses), (b) only examined differences or relations between groups on non-manipulative variables (e.g., examining differences in behavior between males and females, or between students with learning disabilities and at-risk learners: examining the relation of age to reading achievement), or (c) did not report statistical significance of inferential statistics evaluating the effect of the intervention. Using these criteria, over 1,428 relevant studies were screened for eligibility. Out of the relevant literature screened, 130 articles were deemed eligible for inclusion in our review. Data Collection: We imported articles that met the inclusion criteria into Covidence, an online tool for screening and extracting data in systematic reviews and meta-analyses. To analyze the studies, we adapted a codebook from Eekhout and colleagues (2012). We began by randomly selecting five articles not included in our sample, coding two of them together, and then revising the codebook to improve its alignment with our research questions and usability. We then coded the remaining articles independently and discussed discrepancies. We repeated this training process with additional articles not from our sample until we met an a priori criterion of 90% total agreement. Analysis: We will report frequencies of missing data reporting and handling methods in special education group intervention research. We will compare these results to past and current design and reporting standards for the special education field (Gersten et al., 2006; Toste et al., 2023). Results: Results are presently not available. Study coding and analyses are scheduled to be completed by June 2023.
Descriptors: Research Problems, Special Education, Intervention, Educational Research, Research Design
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Information Analyses; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A