NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED656898
Record Type: Non-Journal
Publication Date: 2021-Sep-29
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Effects, Costs, and Cost-Effectiveness of Two Computer Assisted Instruction Math Programs When Used as Supplements to Regular Education Elementary Math Instruction
Matthew E. Foster
Society for Research on Educational Effectiveness
Background/Context: Despite the surge in efficacy studies, a recent meta-analysis indicates that the relative effectiveness of any two math interventions is largely unknown (Nelson & McMaster, 2019). Additionally, policy makers rarely choose educational programs based on an assessment of costs and cost-effectiveness (Hollands et al., 2016; Levin & Belfield, 2015). Given the recent increase in the use of computer assisted instruction (CAI) due to the global pandemic, identifying the effectiveness of CAI math programs relative to one another and examining their cost-effectiveness is an important next step for education researchers. Purpose/Objective/Research Question: To ensure digital equity and provide school personnel and policy makers with empirical data to make informed decisions, the present study evaluated the efficacy of two math programs relative to one another and investigated their cost-effectiveness. Differential response to CAI as a function of participants' numeracy skills prior to receiving CAI was also tested; thereby exploring possible Matthew Effects (cf. Bodovski & Farkas, 2007). Participants in the DreamBox Learning condition were expected to outperform participants in the Zearn Math condition; however, the extent to which DreamBox was more cost-effective than Zearn was unclear as was the potential for Matthew Effects. Setting: Four kindergarten and five first grade classrooms participated in this study, which took place in a large suburban school district in Florida. Population/Participants/Subjects: The sample consisted of 115 students (49% female). Of participating students, 23% were Hispanic (White), 47% were non-Hispanic White, 8% were Asian, 7% were African American, and 10% were multiple races/ethnicities (5% of parents did not report their child's race/ethnicity). At the study's onset, the sample's scores on norm-referenced standardized tests of numeracy (M=128.97; SD=21.63) and nonverbal ability (M=72.93, SD=8.77) were high and below average, respectively. Intervention/Program/Practice: Participants were randomly assigned to receive supplemental CAI with either DreamBox or Zearn. Both programs were adaptive, aligned with CCSSM, approved for use in a local school district and focus on number sense, numbers and operations, equivalence, and place value. Participants interacted with one of the two programs in the presence of intervention specialists for 11 weeks during their math block. Research Design: Pretest-posttest randomized control trial; participants were randomized with equal probability from within classroom to one of the two experimental conditions. Data Collection and Analysis: Proximal to the math programs, children's numeracy skills were assessed during the fall (pre-intervention) and spring (post-intervention) of the school year with the "Research Based Early Math Assessment" (REMA; Clements et al., 2008). Distal to the math programs, the REMA's geometry strand was only administered at post-intervention to protect children's' time away from instruction. Multilevel models were not used in analyses because the ICCs for models for posttest numeracy and geometry were 0.00 and randomizing children from within classroom methodologically controls for the effects of nesting in the 9 classrooms. For proximal math achievement, pretest numeracy scores and group were included as predictors of posttest numeracy scores. For distal math achievement, pretest numeracy scores and group were included as predictors of posttest geometry scores. Each of the prior models was expanded by including the interaction of group by pretest numeracy when predicting posttest scores to examine possible Matthew Effects. Raw scores were used in all analyses and completely standardized results are reported, including R[superscript 2] and Hedges g for each model. The ingredients method (see, for example, Levin et al., 2018; Levin & Belfield, 2015) was applied to the two math CAI programs in order to calculate costs and cost-effectiveness ratios. Local costs were based on the 2018-2019 and 2019-2020 salary schedule for the participating school district. National costs were calculated with the CBCSE Cost Tool Kit (Hollands et al., 2015). Findings/Results: Participants in the DreamBox and Zearn conditions showed statistically significant improvement from pre- to post-test for numeracy and geometry (ps<0.05). Subsequent regression analyses indicated that pretest numeracy (ps<0.001) but not experimental condition (ps=0.15 and 0.77), was a statistically significant predictor of posttest numeracy and geometry. The group of predictors accounted for 70% and 29% of the variance in posttest numeracy and geometry scores. Hedges g suggested that use of DreamBox led to positive gains for posttest numeracy (g=0.32) and geometry (g=0.14). The interaction of group and pretest numeracy was statistically significant interaction (i.e., a Matthew Effect), accounting for an additional 9% of variance for posttest numeracy (but not posttest geometry). As participants' pretest numeracy scores increased, posttest numeracy scores for participants in the DreamBox condition improved more than that of participants in the DreamBox condition with lower pretest numeracy scores and participants in the Zearn condition (see Figure 1). The difference in costs for implementing DreamBox over Zearn was roughly $20 per student (see Table 1 and 2). Sensitivity analyses suggested that the cost-effectiveness of DreamBox improves as the number of students that use DreamBox increases (see Table 2). Conclusions: This study provided tentative support that a relatively low intensity supplemental implementation of DreamBox led to improved math achievement over Zearn. The estimates of Hedges g are consistent with the results of a prior study of DreamBox (Wang & Woodworth, 2011) and Hedges g for numeracy in the present study (0.32) exceeds the What Works Clearinghouse threshold of a 0.25 effect size to be considered of "substantive importance". Evidence of a Matthew Effect is consistent with the cognitive skill-building framework, which predicts that early knowledge of math leads to later achievement because early numerical skills help students acquire math concepts (e.g., Gersten et al., 2009; Watts et al., 2014). This finding also points to a complex interaction between the online learning environment and numeracy proficiency. Consistent with the bioecological model of human development (Bronfenbrenner & Morris, 2006), participants with higher pretest numeracy may have been able to complete more lessons with DreamBox compared to other participants and experienced more positive feedback in the online environment, which resulted in mastering more numerical concepts than their peers. Finally, when costs are compared to outcomes, the results of this study tentatively support adopting DreamBox over Zearn as a supplement to regular education math instruction because DreamBox is more cost effective.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Early Childhood Education; Elementary Education; Kindergarten; Primary Education; Grade 5; Intermediate Grades; Middle Schools
Audience: Administrators; Policymakers
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Florida
Grant or Contract Numbers: N/A