ERIC Number: ED658548
Record Type: Non-Journal
Publication Date: 2022-Sep-24
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Findings from a Switched-Replication Study of the Future Forward Literacy Program
Curtis Jones; Marlo Reeves; Dongmei Li
Society for Research on Educational Effectiveness
Background/Context: Future Forward (FF) is an early elementary literacy program, administered by Education Analytics (EA), that combines one-on-one tutoring with parent engagement to promote student literacy development both at school and at home. In 2011 FF was funded by an i3 grant to develop the program and test its impact in Milwaukee. Two randomized control trial (RCT) studies found the program had positive impacts on literacy, reading achievement, and school attendance (Jones, 2018; Jones & Christian 2021). In 2017, EA received an Education Innovation and Research (EIR) Mid-Phase grant from the U.S. Although the program and its evaluation have been severely affected by COVID-19, three partial studies have been completed examining its impact on reading achievement (Jones, Bower, Pyatigorsky, & Price, 2021, September; Jones & Li, 2022, January) and school attendance (Jones & Li, 2021, March). These studies have found significant impacts on school attendance and achievement (for specific student groups). Program: FF employs a school, community, family partnership approach (Epstein, 2001) to support student literacy development of early primary grade students. The site manager is typically a certified teacher who manages personalized one-on-one tutoring provided by paraprofessionals. The site manager works with the school and tutors to develop a tutoring schedule. This involves identifying times students are pulled out of class to receive tutoring and finding tutors who can work during those times. Students are tutored by the same tutor throughout their time in FF. The site manager provides ongoing support, development, and supervision to the tutors. Each FF student is scheduled for 90 minutes of tutoring each week for one semester. The site manager also works closely with the family engagement (FE) coordinator, who is responsible for family outreach and communication. These typically involve monthly family events and ongoing contacts through in-person meetings, phone call conversations, or email or text conversations. Partially due to the increasing need for support in these schools caused by the COVID-19 disruptions to schooling, the program length was changed from one year to one semester. Current Study: The evaluation used a switched-replication randomized design (Shadish, Cook, & Campbell, 2002), so that all families who requested support would receive FF by the end of the school year, half in the fall and half in the spring. Given the continued impact of COVID-19 on students, we felt that the switched-replication design was the most morally defensible, still allowing for a rigorous impact study but not excluding any students from receiving services. This interim impact study presents the results of FF implementation and measures the impact of the fall semester of FF on students attending these three schools. The purpose of this study is to measure the impact of one semester of FF on students attending these three schools. Specifically..."How was Future Forward implemented in the fall of 2021-22? What was the impact of one semester of the FF intervention on the regular-school-day attendance, SEL, and reading achievement of K-3rd students as compared to BAU reading instruction?" Setting: The 2021-22 program was limited to three (one Wisconsin and two Alabama schools) of the 14 schools originally included in the EIR-funded study (Table 1). The two Alabama schools are located in an urban community. Both schools are relatively high performing, with 51% and 57% reading proficiency, and do not receive Title I funding. They do not provide students Tier II intervention services. Both schools mostly served White students and families before a 2015 consent order changed their attendance areas to include students living in a segregated Black community. The Wisconsin school is in a small town/rural community. The school is low performing, with 32% reading proficiency and receives Title I funding. It serves mostly White (73%), low-income (67%) students. Students do receive Tier II interventions as part of its Response to Intervention process. Participants: Eligible study participants included Kindergarten, first, second, or third grade students without an Individualized Education Plan (IEP) and who were not English language learners. Most of the 127 students were eligible for free or reduced lunch (88%), roughly one-third were Black, and half were White (Table 2). Star Reading was used to calibrate assignment and to obtain balance between assignment groups (Table 3). Local reading assessment results were provided after assignment. These results indicate that students assigned to receive FF in the fall scored higher on local reading assessments than did students assigned to BAU instruction (F = 3.89, p = 0.051). The final analytic sample included 59 BAU and 63 FF students. Only five students attrited (3.9%), three BAU (4.8%) and two FF (3.1%). Research Design: We used a switched-replication design to randomly assign students to either FF or BAU reading instruction; fall BAU students will receive FF in the spring. Assignment was made within 36 classrooms. Reading assessments were administered before assignment in the fall and at the end of participation in January. Teachers completed SEL assessments before and after the fall semester about each participant. Instruments: Star Reading is a short, online adaptive assessment with have high internal reliability (0.95) and concurrent validity with other reading assessments (Renaissance Learning, 2021). Local assessments included the Phonological Awareness Literacy Screening (PALS) (Invernizzi, Swank, Juel, & Meier, 2003) and the DIBELS. Social emotional learning (SEL) was assessed via two scales from the Devereux Students Strengths Assessment (DESSA), Optimistic Thinking and Goal Directed Behavior (Shapiro et al., 2016). Analysis Strategy: We used general linear models (GLM) to estimate the impact of FF using the following equation: is the "outcome (reading achievement, attendance, SEL)" for the i[superscript th] student in the j[superscript th] block, is the impact of FF, and are baseline SEL scores, is the effect of attendance in the months before the study started, is the effect of baseline Star, is the effect of local baseline reading achievement, is the effects of student demographics, and is the effect of block. Dummy replacement method was used for missing baseline data (Puma et al, 2009). Findings/Results: Implementation: Tutoring spanned from October to January. The average student received a total of 25.5 sessions and 2.6 sessions per week (Table 4). All but two students received the targeted amount of tutoring (at least two sessions each week). Family engagement involves substantive interactions or communications with participant family members. Family engagement efforts in Alabama were mostly done virtually, while in Wisconsin family engagement was mostly done in person. The average student's family was engaged successfully 1.5 times per month. Nineteen (30.2%) student families were engaged at least twice per month while 40 (63.5%) were engaged at least once per month (Table 5). Impact: FF had a large, significant impact on Star Reading (B = 0.289). This impact was entirely driven by the differential impact of FF in the two Alabama schools (B = 0.473, p < 0.001). FF was not found to have an impact on school attendance or SEL (Table 6). Consistent with Star Reading results, FF had a statistically significant impact on local reading assessment results in Alabama (B = 0.310, p = 0.012). Conclusions: These findings add to a growing body of evidence showing that FF positively impacts students. In this study, on a limited sample of students and schools, we found a large impact on reading achievement. This is an important result as FF looks to scale up in its next EIR funded grant. Finding that FF remained impactful even while reducing the length of programing suggests this is a viable option for FF moving forward. The shorter duration will reduce costs and increase the students who can be served. The switched-replication study will continue in the spring semester, as former BAU students finish their participation in FF. The evaluation will examine if students assigned to FF in the spring catch up to the students who received FF in the fall. We will also examine the sustained impact of FF one semester after participation ended in the winter. A finding that the assessment results of students assigned to receive FF in the fall and spring are roughly the same would suggest the impact of fall semester FF participation has been sustained through the end of the year.
Descriptors: Elementary School Students, Emergent Literacy, Literacy Education, Tutoring, Parent Participation, Family School Relationship, Reading Achievement, Equal Education, Kindergarten, Grade 1, Grade 2, Grade 3, Reading Programs
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Education; Early Childhood Education; Kindergarten; Primary Education; Grade 1; Grade 2; Grade 3
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Wisconsin (Milwaukee); Alabama
Grant or Contract Numbers: N/A