ERIC Number: ED658665
Record Type: Non-Journal
Publication Date: 2022-Sep-23
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Impact of an EdTech Intervention on Early Primary Learning in Under-Resourced Schools in Malawi Amid Disruptions Due to COVID
Karen Levesque; Sarah Bardack; Abraham Bahlibi; Antonie Chigeda; Symon Winiko
Society for Research on Educational Effectiveness
Background: Malawi's primary education system made progress in increasing access to school over the last decade, but is challenged to provide quality learning in the face of expanding enrollment (World Bank 2021). By Standard 4, 19 percent of students still score zero on Standard 1 math items and only 22 percent are able to comprehend a short reading passage in the primary language of instruction (Ibid.). To address poor learning outcomes, the Malawi Ministry of Education piloted the use of onebillion's onecourse software, an award-winning tablet-based literacy and numeracy curriculum, in about 100 schools and is considering expanding the program nationwide. Initial RCTs on onecourse in Malawi produced significant effect sizes in math (0.63) and literacy (0.42) over 8 weeks and 14 weeks, respectively (Pitchford 2015; Pitchford, Hubber, and Chigeda 2017). A longer 8-month RCT produced significant effect sizes of 0.34 in literacy and 0.15-0.29 in early math (Levesque, Bardack, and Chigeda 2020). While these trials produced significant average effects, absolute gains were modest. Only 9% of the literacy treatment group and 19% of the math treatment group attained emergent or fluent status after 8 months. These studies suggested investigating the impact of longer-term implementations of onecourse. Purpose Originally, we intended to estimate learning impacts of using onecourse for 2 academic years (16 months of intervention) to inform national scaling of onecourse with evidence on a longer intervention period than one school year. However, due to the COVID pandemic, Malawi schools closed multiple times during the study period (see Table 1 for timeline). As a consequence, we adjusted our primary research questions (see underlining): 1. What are the impacts over standard instruction on literacy and numeracy outcomes of using onebillion's onecourse software in Chichewa (either literacy or math) for 40 minutes per day for 2 interrupted school years (totaling 13 months of intervention)? 2. What impact does attendance in the intervention have on learning outcomes? 3. How far do treatment group children progress toward Malawi government benchmarks for reading and math in the lower primary grades? While COVID-related school closures during the study present a unique situation, school disruptions are not uncommon in Malawi. In recent years, schooling has been disrupted by political unrest, a cyclone, and floods, and climate-related disruptions are expected to continue (World Bank 2021). Thus, the results of this study may be relevant for other periods of interrupted schooling in Malawi and in other countries experiencing similar challenges. Program and Participants The onecourse program is called "Unlocking Talent" and provides supplemental learning for Standard 2 children. Figure 1 depicts the intervention. We purposively selected two government primary schools for the study to represent an urban and a peri-urban (more rural) environment and to meet sample size requirements. Ultimately, 578 Standard 2 learners in the two schools were enrolled in the study. The final study sample represented primarily first-time Standard 2 learners of normative age who did not have a severe disability preventing participation. Table 2 shows the gender and age category distributions of the study sample and describes the eligibility criteria. Research Design: The efficacy RCT used a non-clustered, blocked individual random assignment design. Children were assessed at baseline prior to randomization. Independently within each school, children were randomly assigned to two treatment groups (literacy and math) and one control group within four gender (male and female) and age category (6-7 years and 8-10 years) strata. After randomization, we confirmed standardized mean differences on key pre-treatment variables (gender, age category, baseline achievement) satisfied baseline equivalence standards (with or without statistical adjustment), ranging from 0.00 to 0.06 standard deviations in absolute value. Data Collection and Analysis: Primary outcomes for the study were based on the Malawi adaptations of the Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) in the Chichewa language: average EGRA percent correct (literacy intervention) and average EGMA percent correct (math intervention). The same forms of the test were used at baseline (October 2019) and endline (November 2021). We produced two sets of impact estimates: Intent-to-Treat (ITT) estimates representing the impact of being assigned to the intervention, relative to being assigned to the control group; and Treatment-on-the-Treated (TOT) estimates representing the impact of attending the intervention at least 50% of the days the learning center was open, relative to attending the intervention fewer or no days. Further details on the regression model and estimation procedures is provided in Table 3. Findings: At endline, 25 months after baseline, 462 children persisted in the study. Overall attrition was 20%: 18% for each treatment group and 24% for the control group. Standardized mean differences met group design standards for baseline equivalence of the final analytic sample (see Table 4), supporting the causal validity of our impact findings. Treatment groups attained a 64% average attendance rate in the intervention and accumulated an average of 90 hours of time on task. While these participation levels were lower than anticipated, due to COVID-related disruptions (we had targeted 70% attendance and 120 hours of time on task), the resulting impacts were strong. Table 5 summarizes the impacts. Conclusion: Despite COVID-related school closures during the study, and lower than anticipated time on task, onecourse produced statistically significant impacts in both literacy and math. Further, the 13 months of (interrupted) intervention produced larger effect sizes and higher rates of attaining emergent or fluent reading and math benchmarks than the prior 8-month study. Findings suggest that implementing onecourse in both Standards 2 and 3 may help Malawi attain its goals for early grade literacy and math. Further, these impacts were obtained despite disruptions, which may have implications for periods of disruption due to other causes. Nevertheless, pandemic conditions were unique and results for the purposively selected schools do not generalize to all primary schools in Malawi. However, positive results from all four RCTs conducted on onecourse in Malawi to date offer a strong basis for considering program expansion. If undertaken, we recommend a staged approach that evaluates effectiveness at larger scale and further investigates 2-year program impacts under less unusual circumstances.
Descriptors: Foreign Countries, COVID-19, Pandemics, Elementary Education, Barriers, Computer Software, Educational Technology, Literacy Education, Mathematics Education, Program Effectiveness, Reading Achievement, Mathematics Achievement, School Closing, Effect Size, Reading Skills, Mathematics Skills, Primary Education
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Education; Early Childhood Education; Primary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Malawi
Grant or Contract Numbers: N/A