ERIC Number: ED656939
Record Type: Non-Journal
Publication Date: 2021-Sep-29
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Impact of Cognitively Guided Instruction on Elementary School Math Achievement: Five Years after the Initial Opportunity
Robert Schoen; Chris Rhoads; Amanda Tazaz; Walter Secada; Alexandra Stone
Society for Research on Educational Effectiveness
Billions of dollars are spent every year on teacher professional development (PD) programs (Fermanich, 2002; Odden et al., 2002; TNTP, 2015; U.S. Department of Education, 2014). The few rigorous evaluations of PD programs that have been done usually report a null effect on student achievement, leading many scholars to conclude that teacher PD programs do not have the intended effect (Garet et al., 2016; Gersten et al., 2014). Teacher PD programs based on Cognitively Guided Instruction (CGI) have been the subject of several experimental studies that evaluated the effect of CGI programs on student achievement (Carpenter et al., 1989; Jacobs et al., 2007; Schoen et al., 2018; 2020). Results have been mixed, but overall positive. Purpose/Objective/Research Question: The present study investigates the long-term effect of the opportunity to participate in a multiyear CGI PD program. Specifically, we estimate the effect of school-level randomization on school mathematics achievement after five continuous years of the availability of the opportunity. The following research questions (RQs) guided the study. RQ1: What is effect of the opportunity for elementary mathematics teachers to participate in CGI professional development on grades K-2 student achievement during the 2017-2018 school year as measured by the i-Ready Diagnostic? RQ2: What is effect of the opportunity for elementary mathematics teachers to participate in CGI professional development on grades 3-5 student achievement during the 2017-2018 school year as measured by the i-Ready Diagnostic and the Florida Standards Assessment? RQ3: To what extent does the effect on mathematics achievement vary as a function of student: grade level, race/ethnicity, disability status, English-learner status, or gifted status? Intervention/Program/Practice: The CGI PD program was designed and delivered by Teachers Development Group under the direction of Linda Levi, a coauthor of the three definitive CGI books (Carpenter et al., 1999; Carpenter et al., 2003; Empson & Levi, 2011). There is a separate program for teachers of grades K-2 and 3-5; each program is designed for teachers to participate for three years. Research Design: This study used a blocked, school-randomized design with 22 schools assigned to an intervention or comparison condition during spring 2013. Data analysis used all available data for students in randomized schools during the 2017-2018 school year. Setting: The study occurred in 22 public schools in two adjacent school districts in Florida. Teacher participation in the program was neither required nor funded by the school or school district. The CGI PD program was available for teachers in the schools assigned to the intervention condition for a five-year period. In the first two years of the study, eligibility was limited to grades 1 and 2 teachers and instructional support personnel. During the subsequent three years, all personnel in the schools were eligible to participate (including personnel in schools randomized to the comparison group). Table 1 shows that substantially more teachers in schools randomized to the intervention condition participated in CGI PD. Population/Participants/Subjects: Table 2 provides a description of the students in the sample by condition. Across all grade levels, there was at least one valid mathematics achievement test score for 11,226 students within 14 schools in District 1 and 5,693 students within 7 schools in District 2. One school in the comparison condition was closed after the second year of the study period, so that school is not included in the analytic sample for 2017-2018. Data Collection and Analysis: Participating school districts provided data from district records. Both districts administered the i-Ready Diagnostic (Curriculum Associates, 2018) three times per year to all students in grades K-5 during the 2017-2018 school year. All students in grades 3-5 took the Florida Standards Assessment (FSA; FDOE, 2018) in Spring 2018. Results are based on the total math score from each of the two respective tests, both of which are vertically scaled across grade levels. Impact estimates were obtained using hierarchical linear models. Since there were three i-Ready scores available, models involving i-Ready were three-level HLM models with measures nested within students nested within schools. For FSA outcomes, there are only two levels of nesting. Additionally, models included a school-level indicator for treatment status, the school level average fall i-Ready score for Kindergarteners (fall i-Ready scores for other grade levels are not exogenous to treatment in 2017-18), dummy variables for blocks, and indicators for student grade and student FRL, ELL, gifted and disability (SWD) status. Findings/Results: Results pertaining to RQs 1 and 2 can be found in Table 3. The effect-size estimate for grades K-2 was positive, small, and not statistically significant. The effect-size estimates obtained for the two grades 3-5 measures (i.e., i-Ready, FSA) were both 0.16 and, therefore, fall in the medium range for experimental studies according to Kraft (2020). The estimated effect on i-Ready for grades 3-5 was statistically significant, but the FSA estimate was not. Analyses conducted for RQ3 examined the interaction between treatment and student-level factors. Table 4 looks at grade level as a moderator separately for each grade band (K-2 and 3-5) and outcome variable (i-Ready and FSA). Table 5 combines data across grade K-5 and treats grade level as a continuous moderator for the i-Ready outcome. Results indicate larger effects at grades 4 and 5 relative to grade 3 as well as a generally increasing trend in effects for higher grades. Table 6 presents results of analyses exploring the role of student demographic characteristics as moderators. There is little to no evidence of treatment effect variation as a function of most of the moderators examined. Conclusions: Despite only partial participation of teachers in schools randomized to the intervention condition, and some participation of teachers in schools randomized to the comparison condition, effects of the initial randomization on grades 3-5 students were observed five years after randomization. The overwhelming majority of teachers that participated were early elementary teachers, which makes the results of the grade-level moderator analyses particularly interesting. Space limitations preclude a full discussion of the implications of these results. Suffice to say that they provide interesting insight into the potential long-term effects of a math PD program for elementary teachers.
Descriptors: Teaching Methods, Mathematics Instruction, Instructional Effectiveness, Elementary School Students, Mathematics Achievement, Faculty Development, Program Effectiveness, Mathematics Teachers, Elementary School Teachers, Instructional Program Divisions, Racial Differences, Ethnicity, Students with Disabilities, English Language Learners, Academically Gifted, Student Characteristics, Effect Size
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: Florida
Grant or Contract Numbers: N/A