Background: Early math knowledge is the strongest predictor of later academic achievement (Duncan et al., 2007), but math achievement gaps that are present during preschool prior to kindergarten entry (Starkey & Klein, 2008) persist in elementary school. This is especially true for young children who are very low-performing in math relative to their peers and at-risk for math difficulties. Longitudinal studies show that children who start low in math continue to struggle throughout the elementary grades (Jordan et al., 2006). This issue has been amplified by the pandemic where young children with disabilities and those at-risk for learning difficulties were identified and supported at lower rates (Barnett & Jung, 2021). Thus, there is a critical need to identify effective interventions to support preschool children with low math knowledge who are at-risk for mathematical learning difficulties. Purpose: The purpose of this IES-funded project is twofold. The first is to conduct a replication study of Pre-K Mathematics Tutorial (PKMT), a Tier-2 tutoring intervention for improving the mathematical knowledge of at-risk prekindergarten (pre-k) children. The original efficacy study, which met WWC standards without reservations, found that PKMT had statistically significant positive effects on math outcomes for very low-performing children (Barnes et al., 2016). The second purpose is to systematically vary the implementation of PKMT to compare the relative effectiveness of two distinct delivery models - a pull-out model and an in-class model. Both models were tested in randomized studies. The first study (2022-23) involved a "pull-out" model where tutors worked with pairs of children outside the classroom, and the second study (2023-24), currently underway, involved an "in-class" model where classroom teaching staff work with pairs of children in the classroom. For both randomized studies, we hypothesized that math outcomes would be higher at pre-k post-test for children in the treatment group than for children in the control group. This paper presents findings from the study of the pull-out delivery model which was the way PKMT was delivered in the original efficacy study. Setting: The study took place in one large Head Start program and two midsize state preschool programs in Sacramento County, CA. Participants: Our sample was comprised of 216 (112 T; 104 C) ethnically and linguistically diverse preschool children (mean = 4.48 years) eligible for Head Start or state preschool based on family income. About half the sample were girls. Latinx children were the largest group at 35% with African Americans (18%), Asians (14%), and White (17%) making up most of the remaining sample. See Table 1 for the breakdown by condition. Intervention: The PKMT intervention components include: a) curriculum materials (a scope and sequence, 20 scripted lessons with concrete manipulatives, progress monitoring tools); b) an intensive delivery model where tutors work with children outside of the classroom for 15 minutes 4-5 times per week; c) a rigorous training process for tutors that includes four days of professional development (PD) workshops; and d) fidelity of implementation monitoring (a fidelity of implementation form and fidelity support visits by an early intervention coach). All training and implementation procedures followed those that were used in the original study. Research Design: We employed a cluster randomized controlled trial to evaluate the impact of PKMT on children's math outcomes. The unit of random assignment was the pre-k classroom. The resulting design was multi-level with children who met the screening criterion nested within pre-k classrooms which are located on school sites. Data Collection and Analysis: Assessors were trained and certified on two early math measures, the Child Math Assessment (CMA: Milburn, Lonigan, DeFlorio & Klein, 2018) and the Test of Early Mathematics Ability (TEMA-3 Ginsburg & Baroody, 2003). Assessors screened all four-year-old children in participating classes on a measure comprising three tasks from the CMA (see Barnes et al., 2016). Children who met the screening criterion for very low performance in math and had parental consent were included in the research sample, and assessed individually on both math measures at pretest and again at post-test. Attrition at post-test was low (7-8%). The independent evaluator assessed baseline equivalence and estimated impacts on each math outcome measure for the Pull-Out vs. Control contrast. For the impact analyses, each math achievement outcome (measured at Wave 2) was regressed on a study group indicator and the math achievement pretest (measured at Wave 1) using a three-level hierarchical linear model in which children were nested within classrooms which were nested within randomization blocks. Hedges' g was used to compute effect sizes. The baseline equivalence analyses were conducted using the same three-level hierarchical linear model structure and regressed each Wave 1 math pretest on the study group indicator. Findings/Results: Baseline equivalence analyses established that any differences between the treatment and control groups were not significantly different from zero. Impact analyses for the Pullout vs. Control contrast showed a substantively important and statistically significant impact on both math measures, the CMA (ES = 0.35, p = 0.006) and the TEMA Raw Score (ES = 0.37, p = 0.003). Figures 1 and 2 illustrate these baseline and impact findings for each math outcome measure. Conclusions: Our findings demonstrate that we successfully replicated the efficacy of the Pre-K Mathematics Tutorial intervention, using the pull-out delivery model, in different programs in a different county and in the context of the pandemic. Although significant and meaningful, we noted that the gains on the CMA were not quite as large as they were in the original study. Given that the tutoring intervention was implemented with very high fidelity (mean =0.96), and the pretest scores were comparable in both studies [Original: 0.26 (T), 0.26 (C); Replication: (0.26 (T), 0.24 (C)], we think this difference is likely due to absenteeism and generally poorer attendance in pre-k settings since the pandemic. As students in our sample were absent more frequently than in the original study, there was a greater need for make-up sessions. In the presentation, we will discuss how absenteeism may have disrupted children's learning over the intervention year.