ERIC Number: ED657021
Record Type: Non-Journal
Publication Date: 2021-Sep-28
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Testing the Impact and Scalability of the Future Forward Literacy Program
Curtis Jones; Kate Bauer-Jones; Dongmei Li
Society for Research on Educational Effectiveness
Future Forward (FF) is an early elementary literacy program that pairs one-on-one tutoring with parent engagement. In 2017, Education Analytics (EA) received an Education Innovation and Research (EIR) Mid-Phase grant to expand FF to 14 schools in seven school districts across three states. The randomized control trial (RCT) study of the 2019-20 FF program was designed to examine the effectiveness of its scale-up to 14 schools and its impact on reading achievement, social-emotional learning (SEL), and school attendance. Within FF program sites, the manager, who is typically a certified teacher, supervises personalized one-on-one tutoring provided by paraprofessionals or volunteers. The family engagement (FE) coordinator, who is typically a community member or parent from the school, is responsible for family outreach and communication. These typically involve monthly family events and ongoing contacts. In the spring of 2020, schools were shut down nation-wide because of the COVID-19 pandemic. This prevented analyses of reading achievement and SEL. However, the study was still able to measure the impact of FF on school attendance and use baseline measures of achievement and SEL as covariates. Research Questions: We explored the following three questions: (1) How well was FF implementation scaled up to 14 schools across three states? (2) What was the impact of FF on regular-school-day attendance compared to students receiving Business-as-Usual (BAU) literacy instruction? (3) Did FF have a differential impact on student subgroups? Evaluation Methods: In the fall of 2019, 293 students were randomly assigned to FF and 294 to BAU. Assignment occurred within blocks (grade levels x schools). With three grade levels, 14 study schools, and two schools without Kindergarten, there were 40 blocks. Assuming a fixed program effect, and 40% of variance explained by covariates, the study, prior to attrition, had an 80% likelihood of detecting a 0.184 standardized impact. Seven study schools served rural communities and seven served urban communities (Table 1). Modeling Strategy: Generalized linear models (GLM) with linear error terms and an identity link function were used to estimate the impact of FF on attendance. Both baseline and follow-up attendance, baseline reading achievement, and baseline SEL were standardized within grade levels. We use robust standard errors and fixed block effects (Figure 1). We also conducted two robustness checks. 1. A simple model including fixed block effects and FF participation, and 2. The full model with clustered error terms instead of fixed block effects. Attrition and Sample Characteristics: During the academic year, 20 students (3.4%) exited the study. Eight BAU (2.7%) and twelve FF (4.1%) students attrited. The combination of overall and differential attrition meets WWC (2020) standards. Tables 2 and 3 present demographic characterizes of BAU and FF groups. Program Implementation Results: The average student across the schools received 2.6 tutoring sessions per week. Of the 281 FF participants, 252 (89.7%) received the full tutoring program (at least two sessions a week) (Table 4; Figure 2). Given that 11 out of 14 sites met the implementation benchmark, EA successfully scaled up the tutoring component of FF (Figure 3). The FE component of FF involves substantive interactions or communications with participant family members. Two or more family contacts each month were required for a student to receive FE as intended. Only 30% of students' families were successfully contacted by FF staff at least two times per month (Table 5; Figure 4). None of the 14 sites met the 80% threshold requirement for full implementation (Figure 5). Impact Results: Table 7 presents the unadjusted follow-up attendance results for students retained in the study: attendance was greater for FF students during the tutoring period by 1.4 percentage points (M = 91.2% vs. M = 89.8%). On average, FF participants missed one fewer day of school. FF had a 0.13 standardized impact on student attendance (Table 7). This standardized impact corresponds to a 1.3 percentage points higher attendance rate for FF students. This impact was robust with a simple model that only controlled for fixed block effects ([beta] = 0.14, SE = 0.07, p = 0.047) and in a model which clustered error terms within blocks ([beta] = 0.14, SE = 0.06, p = 0.029). The impact of FF was larger with Black ([beta] = 0.18) and male ([beta] = 0.27) students (Table 7). The model-adjusted Black FF participants' attendance rate was 1.9 percentage points higher than that of Black BAU students. For male students, participating in FF resulted in a model-adjusted 2.8 percentage points better attendance rate. Across gender and race, students with low school attendance prior to the start of the program also bene ted more from their participation ([beta] = 0.25). Black male students were especially affected by their participation in FF (Table 7); their school attendance rate was a model-adjusted 4.3 percentage points higher ([beta] = 0.42). Although only marginally significant, the impact of FF on Black male students with low school attendance at the start of the FF program was large ([beta] = 0.59) (Figure 6; Figure 7). Summary: The EIR-funded RCT study from the 2019-20 school year, adds to the growing body of evidence of the scalability and effectiveness of the FF program. Regarding scalability, most students received tutoring as intended. Regarding FE, only a minority of students' families (30.2%) received FE as intended. Sites struggled to connect with participant families at the expected level. The current study's FE implementation was still much improved from the implementation during the 2018-19 school year, where only 6.6% of students received the intended amount of FE (Jones et al., 2020). Because in-person schooling stopped due to the COVID-19 pandemic, our analyses were limited to school attendance. Regarding attendance, FF was still found to have a statistically significant, positive impact. This is the second study of FF to report a positive effect on attendance (Jones & Christian, 2020), which further validates the FF model design pairing tutoring with parent engagement to support the literacy development of students and families.
Descriptors: Emergent Literacy, Elementary School Students, Literacy Education, Tutoring, Parent Participation, Reading Programs, Program Effectiveness, Program Implementation, Attendance Patterns
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A