ERIC Number: ED657112
Record Type: Non-Journal
Publication Date: 2021-Sep-28
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Supporting Teachers in Argument Writing Instruction at Scale: A Replication Study of the College, Career, and Community Writers Program (C3WP)
Nicole Arshan; Cj Park; Allison Milby; Rebecca Goetz
Society for Research on Educational Effectiveness
Background: College- and career-ready standards cite the "special place of argument writing" in preparing students to be college students, professionals, and engaged citizens in an information-rich society (NGA, 2010). To support these critical skills, the National Writing Project (NWP) developed the College, Career, and Community Writers Program (C3WP). An initial RCT of C3WP found positive impacts on grade 7-10 student source-based argument writing (authors, 2017). This study sought to replicate the impacts of 1 year of C3WP on secondary students (in 2018-19) and estimate the impacts of a second year of C3WP for secondary students and a first year of C3WP on upper elementary students (in 2019-20). The COVID-19 pandemic interrupted implementation of the program and prevented the collection of student outcomes in spring 2020. We therefore have student impact data only for the replication study. This study extends our knowledge in several ways. First, it replicates the original evaluation. Second, it provides evidence of effective teacher professional development (PD) at a scale unusual in the literature. Finally, the teacher impact estimates provide preliminary data on C3WP's effectiveness in upper elementary grades. Research Questions: (1) What is the effect of C3WP on students' writing achievement? (2) What is the effect of C3WP on teachers' writing instruction? and (3) How did sites and districts implement C3WP? Program: C3WP has three components: (1) instructional resources designed to develop students' argument writing skills; (2) a formative assessment tool and (3) intensive PD to support program implementation. C3WP is implemented in cycles. Teachers plan implementation of instructional resources in PD. Teachers then implement the resource in their classroom, often with coaching support from a PD provider. Finally, teachers bring the writing generated by the resource to the next PD session, where they analyze it using the formative assessment tool, launching them into the next cycle of instruction. NWP expects teachers to receive at least 40 hours of PD and implement at least 4 instructional cycles a year. Research Design: This study is a clustered, district-randomized controlled trial with a business-as-usual comparison condition. Sample: The study occurred in 47 rural, high-needs rural school districts in 16 states. The secondary teacher sample includes ELA teachers in grades 7-10 in 2018-19 and 2019-20. The upper elementary teacher sample includes teachers responsible for ELA instruction in grades 4-5 in 2019-20. Students are included in the sample if they completed the grade 7-9 baseline assessment in spring 2018. This assessment was administered in the first five weeks of the school year. Student and district overall and differential attrition meets the What Works Clearinghouse's optimistic boundary for attrition. Data and Analysis Implementation data came from administrative data sources and a teacher survey. The Year 2 survey also asked C3WP teachers to describe program impacts. These responses were coded inductively and analyzed for broad themes. To measure teacher outcomes we administered an instructional log two weeks each year of implementation. The log asked teachers short, low-inference questions about instruction (e.g., time spent on writing). We also administered surveys to understand the difference in professional development offered to C3WP and control teachers in Year 1 (we did not administer surveys to control teachers in Year 2). To measure student outcomes we administered on-demand, source-based, argument-writing assessments fall 2018 (baseline) and spring 2019 (outcome) to grade 7-9 students. Students' writing was scored using the Analytic Writing Continuum for Source-Based Argument Writing (AWC-SBA). Inter-scorer reliability was above 80% on each attribute. We estimated impacts using hierarchal linear models (HLMs) to cluster data as appropriate (eg, students within districts). Models include a vector of randomization block fixed effects to account for randomization within blocks. Final instructional log analyses (in progress) will include interaction terms to estimate differences in teacher practice outcomes by year and semester. We will also use instrumental variable analyses to investigate the extent to which measured shifts in Year 1 instructional practice mediated student impacts. Findings: Overall, most teachers received at least 40 hours of PD in Year 1, though not Year 2, which was interrupted by COVD-19. Most treatment teachers (83%) completed four cycles of instruction in Year 1. Just 61% of teachers in Year 2 completed four cycles, although most teachers (93%) completed at least three. Teachers primarily used the instructional resources that developed foundational skills (e.g., writing and revising a claim). In Year 2 a relatively larger percentage of grade 7-10 teachers used advanced materials, including researching self-selected topics. In grade 4-5, a few teachers reported progressing to a self-directed argument, despite having less than one year of C3WP implementation. Preliminary impact and treatment-control contrasts find that (see appendices B-D): C3WP had a positive and statistically significant impact on all 4 attributes of student writing measured. Estimated impacts (g = 0.18-0.19) are similar in size to those in the initial study being replicated. C3WP had positive and statistically significant effects on some measured aspects of teacher practice. C3WP teachers received substantially more PD than control teachers Conclusions: This study is one of only a few large-scale RCTs replicating positive impacts of teacher PD on student outcomes at scale (see, e.g., Bouley et. al., 2018). Further, rigorous studies demonstrating student impacts are often conducted with small samples, meaning that estimated effects may not be replicable in new contexts or at scale (Schneider, 2018). In contrast, the C3WP sample in this study included 108 schools in 47 districts across 16 states. To contextualize this scale, the 2016 WWC practice guide on secondary writing includes 15 studies (Graham, et. al, 2016). One is the original C3WP study replicated here. The other 14 studies are set across approximately 98 schools. C3WP features an effective PD structure that could be adapted to other subjects. Despite the program's focus on di cult skills, teachers describe students as engaged and motivated in the work. Finally, the program is effective in developing a skill critical for success as college students, professionals, and engaged citizens. As such, the findings we present will have important implications for practitioners, policymakers, and program developers.
Descriptors: Writing Instruction, Persuasive Discourse, Program Effectiveness, Secondary School Students, Elementary School Students, Intermediate Grades, Writing Achievement, Program Implementation, Rural Schools, Disadvantaged Schools, Language Arts, English Instruction, Secondary School Teachers, Grade 7, Grade 8, Grade 9, Grade 10, Grade 4, Grade 5, Faculty Development, Writing Evaluation, School Districts, National Programs
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Related Records: EJ1408303
Publication Type: Reports - Research
Education Level: Secondary Education; Elementary Education; Intermediate Grades; Grade 7; Junior High Schools; Middle Schools; Grade 8; Grade 9; High Schools; Grade 10; Grade 4; Grade 5
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A