Enhancing science performance of middle-school students with and without developmental and behavioral-based disabilities using the Content Acquisition Podcast Professional Development approach

1.1 The role of vocabulary within the science classroom and the need for explicitness

The National Research Council (NRC) emphasizes student engagement in inquiry-based investigations with rich opportunities to communicate and collaborate with their peers in order to learn science content (2012). For students to communicate ideas effectively in the classroom, participate in investigations, and be successful in fully understanding content, it is necessary to learn and understand underlying science terms and concepts (Parsons & Bryant, 2016). However, many terms used within the science classroom are not commonly used in students' everyday lives.

Being able to understand the meaning of a scientific term or phrase and apply it toward science content is an expectation within the Common Core Standards (Scruggs et al., 2013). For example, one of the English Language Arts standards for science and technical subjects state that students must be able to “determine the meaning of symbols, key terms, and other domain-specific words and phrases as they are used in a specific scientific or technical context…” (Common Core State Standards, 2020). To meet this expectation, students must understand the meaning of terms such as “convection,” “sublimation,” or “phenotype” and apply it to the science lessons and investigations. Cervetti et al. (2012) suggested that multiple exposures to a concept or word could enhance the student's knowledge allowing them to better understand new, more complex information. Additionally, students who had literacy instruction embedded within science curriculum understood the big ideas better and could communicate these concepts using the new vocabulary terms in their writing (Cervetti et al., 2012). As such, to provide effective instruction, science teachers should utilize evidence-based science and literacy practices (Block, 2020; Wellington & Osborne, 2001).

1.2 Impact of developmental and behavioral-based disabilities on science vocabulary acquisition

The achievement discrepancy between students with and without DB disabilities could be attributed, in part, to the highly technical vocabulary used in the science classroom. Specifically, science includes a large number of specialized vocabulary words, some of which have alternative meanings in other contexts, such as revolution or solution (Mason & Hedin, 2011; Wellington & Osborne, 2001). For this population of students, the processing speed and accuracy required to acquire numerous pieces of complex information in rapid and often overlapping succession, while attempting to sort and store content within working and long-term memory is simply not realistic if specially designed scaffolds and instructional approaches are not in place (Swanson, 2015; Swanson et al., 2009).

Whereas general education students often develop and use memorization strategies to learn word meanings (Harmon et al., 2005; Songer & Linn, 1991), students with DB disabilities often lack effective strategies to learn vocabulary words (Harmon et al., 2005). As such, these students require vocabulary instruction that is both explicit and systematic to provide the firm foundation necessary to learn and then apply new vocabulary knowledge to scientific problems (Kaldenberg et al., 2015; Therrien et al., 2011). We detail specific evidence-based vocabulary practices for this population in the next section.

An understanding and appreciation of the learning needs of this population of students are needed considering the variability of the effectiveness between some prevailing teaching methods. An example is students with specific learning disabilities in the area of reading. These students often face breakdowns in how their working memory takes in auditory and visual information, activation of schemas in long-term memory, and forging new learning connections. In plain terms, if the teacher mentions the definition of a new term orally, and then quickly moves on to discuss its relationship to other terms and concepts, the student with a learning disability may be sitting confused trying to picture and integrate just the new definition with existing schemas in long-term memory while having given no active cognitive processing to the additional information provided by the teacher (Kennedy & Romig, 2021). As students get older and realize this cycle of frustration and academic failure, they can become deeply disengaged from learning with additional unwanted outcomes (Blondal & Adalbjarnardottir, 2012; Finn, 1989). Table 1 presents a brief overview of students with DB disabilities and their unique challenges learning vocabulary in science.

Researchers have investigated ways to enhance the science vocabulary of students with DB disabilities. For example, Knight and colleagues (2012) investigated whether the incorporation of explicit instruction within science lessons would enhance the science vocabulary knowledge and comprehension of students with autism and intellectual disabilities. Specifically, they included strategies such as the model-lead-test approach, and providing examples and nonexamples. They found that using explicit instruction strategies increased the number of correct responses related to the targeted terms as well as supported generalizing their knowledge about these terms in inquiry-based investigations. Cavanaugh and colleagues (1996) investigated the use of response cards to support the science vocabulary knowledge growth of students with learning disabilities, emotional behavioral disabilities, and intellectual disabilities. They compared the use of response cards to having the students learn the words passively. They found that when students were more actively involved in learning the science vocabulary, they performed higher on the science vocabulary tests than when they learned about terms passively.

1.3 Content Acquisition Podcast Professional Development process

To assist science teachers in providing evidence-based vocabulary instruction during lessons, Kennedy et al. (2017) developed the CAP-PD process. The CAP-PD was developed to help support inclusive science teachers' knowledge of and readiness to implement the aforementioned evidence-based vocabulary practices. CAP-PD consists of three components: (a) asynchronous multimedia instruction that builds teacher content and procedural knowledge of target skills through multimedia modeling presentations for teachers (see https://vimeo.com/448122821 for a sample); (b) CAP-S that teachers can use during instruction, which include evidence-based practices that teachers were exposed to within modeling videos (see www.vocabsupport.com for samples); and (c) individualized feedback provided through email based on live classroom observations using the Classroom Teaching (CT) Scan, a descriptive observation tool used to capture teachers' use of instructional practices (Kennedy et al., 2017). We detail the CT Scan and its functionality along with notes on the subsequent coaching in the methods section.

1.4 Study purpose

2.1 Sample

Participants included 980 sixth-grade students drawn from 13 inclusive science classrooms spread across four rural school districts in the United States. Demographic information originated from teacher records. Specifically, teachers assigned anonymous codes for their students that included student demographic information. Researchers had access only to the anonymous codes whereas the teachers maintained the master list, which connected the student to the code. After receiving university and district consent, families were informed of the study and were given the option to opt their child out of the data collection activities. Of these students, 51.3% were male and 48.7% were female; 73.4% of the students were White, 6.3% were Latinx, 16.2% were Black, 2.6% were Asian, and 1.53% identified as being of two or more races. Approximately 67% of students were eligible for free/reduced lunch. Of the participants, 12% were identified as having a DB disability, broken down by disability type in Table 2. Although it is possible for some of the students to have had more than one DB disability, students in this study were classified by the primary DB disability identified in the Individualized Education Plan. Among students with a DB disability, 45% had a teacher who participated in CAP-PD (N = 52) and 55% had a teacher who did not participate in CAP-PD (N = 64). Approximately 47% of students without a DB disability had a teacher who participated in the CAP-PD (N = 405) and about 53% of students without a DB disability had a teacher who did not participate in the CAP-PD (N = 459). More information about the student demographics by teacher is available in Table 3.

The participating students came from six middle schools representing four school districts in rural areas of the Mid-Atlantic region. The schools had an average of 564.7 students. All of the 13 participating teachers were White, with the majority (69.2%) identifying as women. Educationally, the highest degree earned by 53.8% of the sixth-grade teachers was a Master's degree, and, on average, the participants had taught for 13.3 years.

2.2 Measures

2.3 Design

We employed a randomized controlled trial in which 13 sixth-grade science teachers were randomly assigned to two groups: six teachers were placed in the CAP-PD intervention group, and seven teachers served as the comparison group, using their typical vocabulary instruction methods. Regardless of the assigned condition, each participating teacher was observed six times with their instructional moves recorded using the CT scan. The purpose of the first three observations was to create a baseline of teacher vocabulary instruction, whereas the latter three were to determine the duration of vocabulary instruction and the frequency of evidence-based vocabulary practices used post-intervention.

Baseline observations occurred during the first 3 weeks of the school year. To reduce bias, researchers were blinded to which teachers were in the CAP-PD and control conditions. In the control condition, teachers used their usual practices. Researchers wrote coaching emails following baseline observation number three for all teachers (see details below). The emails were sent to an independent third party who maintained the list of teacher assignments and either forwarded or withheld the emails. Teachers in the CAP-PD condition received access to a library of CAP-S slides and the CAP-TV library with instructions on how to use each. Teachers were instructed to watch each CAP-TV at least once, and then were referred to watch the videos additional times within coaching reports (see below). Some of the evidence-based vocabulary practices teachers received instruction on included providing student-friendly definitions, examples and nonexamples, word parts, and demonstrations. Teachers in the control condition received nothing from researchers other than communications to schedule observations and assessments. Following baseline, the teachers in the treatment group received the CAP-PD. The researchers scheduled to observe each teacher from both groups once per month across the fall semester. Observers asked teachers to teach at least some vocabulary on days they agreed to be observed, and the researchers stayed for the entire lesson (typically 50 minutes). The CT Scan observation tool was used to document what practices were used and for how much time.

The CT Scan is a low-inference observation tool that allows an observer to record what practices the teacher used in real time. It is available free within the COACHED web app (www.coached.education.virginia.edu) and a labeled screen shot is contained in Figure 1. The CT Scan allows the observer to document the broad category of practice (e.g., vocabulary, classroom management, science inquiry), the specific practice (e.g., student-friendly definition, example), and implementation markers of quality associated with each practice (e.g., uses cues, clear language, reviews background knowledge, uses images, elaborates on definition, confirms understanding). The observer and teacher use the time feature of the CT Scan to record how much time was spent using certain practices. For instance, some of the evidence-based vocabulary practices tracked in this study included providing student-friendly definitions, examples and nonexamples, opportunities to discuss the term meaning, and highlighting parts of the words. Observers can also record the specific term or topic being taught, what students were supposed to be doing (e.g., taking notes, answering questions), the group size (e.g., whole, small, individual), any visual aids being used (e.g., pictures, graphic organizers, object), and take qualitative notes. Finally, they can record each question the teacher asked, feedback statements, and other countable data. All data recorded in the CT Scan align with the time they occur and overlay to give an unbiased and comprehensive look at the lesson without needing to assign a quality score. The quantitative data gathered “speak for themselves” and can be used to generate coaching and feedback reports.

The coaching emails were written using data from the CT Scan following each observation. Figure 2 has a labeled screen shot of the feedback form. The feedback notes were tied to the specific implementation markers either observed or not for each practice. For example, if the teacher used clear language when delivering a student-friendly definition, the observer might write “using clear language is critical for students with DB disabilities because you don't want to use other words they may not know. Clear language gives them the best chance for learning success. You did an awesome job today being super clear with your language—keep it up!” If the teacher did not use images, the observer might write “I didn't see you using images to support your students' learning when doing vocabulary today. This is really important because many students do not have mental images already in place to visualize new terms or concepts, and providing them with one can really make a difference. If you use the CAP-S slides we provided, you will find a trove of images. You can also see a teacher modeling use of vocabulary with images at the 7:30 mark of the CAP-TV for this practice. I'd love to see you using images during my next visit.” This process would repeat for each practice the teacher used during the lesson. The feedback would also note total numbers of opportunities students had to respond, feedback statements, and a range of other data. For a more detailed explanation of the feedback process, please see Kunemund et al. (2021, 2022).

2.4 Data analysis

Analysis of variance (ANOVA) was conducted to assess whether significant differences were present between students with and without DB disabilities in classes where teachers participated in the CAP-PD and those who did not on their MOSART post-assessment scores. First, a two-way factorial ANOVA was conducted, which included DB disability status and access to a teacher who participated in the CAP-PD serving as independent variables. The homogeneity of variance was analyzed, and the effect size was determined using the Cohen's d statistic. Since, DB disability status and teacher CAP-PD participation consisted of two groups each, in order to analyze differences between groups, a new variable was created, which coded the participants into one of four groups (i.e., students without DB disabilities whose teacher participated in CAP-PD, students with DB disabilities whose teacher participated in CAP-PD, students without DB disabilities whose teacher did not participate in CAP-PD, and students with DB disabilities whose teacher did not participate in CAP-PD). A one-way ANOVA was conducted with this variable to also be able to investigate differences between groups on the MOSART posttest scores using post hoc testing.

It was decided to use ANOVAs instead of multilevel modeling for the first question in order to provide a simpler initial analysis of differences between the groups focused upon in this study. In addition, student participants were recoded so that each group are given their own unique code based on their DB disability status and teacher CAP-PD participation. This allowed for conducting a one-way ANOVA post hoc test to analyze differences between groups.

As with most school-based research, students were nested within teachers' classrooms, which were nested within schools. There are three levels of nesting in this study: (a) students (Level 1), (b) class (Level 2), and (c) teacher (Level 3). The students form the foundation of this model as the Level 1 variable. The students belong to specific classes (Level 2), and a teacher (Level 3) provides instruction to a set of classes. In total, at Level 2 there were 48 classes, and at Level 3 there were 13 participating sixth-grade teachers. The participating teachers taught multiple classes each. Random assignment to create a treatment group, while underpowered, occurred at Level 3. All of the classes taught by a particular teacher received the same type of instruction, and likewise all of the students within those classes received the same instruction. At Level 1, multiple covariates (DB disability status, English proficiency, MOSART pretest scores, and average CBM scores) were included. These variables were centered as the students (and their related characteristics) should be tied to their specific classes (Level 2). At Level 2, the classrooms were centered around the teacher means (Level 3). This prevented confounding these levels of the model and allows for only within variance to remain. At Level 3 the treatment variable (group) was included. Student DB disability status and group membership were dummy coded. The variables of English proficiency, MOSART pretest scores, and, for one of the questions, average CBM scores were included in order to control for the influence these factors may contribute to the association between the MOSART posttest scores and the students' DB disability status and whether their teacher participated in the CAP-PD. Without controlling for the influence of these variables, they may skew the degree of significance of the associations. For instance, it can be challenging for students learning English who are still developing their knowledge of academic and science-specific vocabulary in English to learn science without support (Garcia, 2005; Terrazas-Arellanes et al., 2018). As such, the multilingual variable was included to mitigate the potential influence it may have on the association between DB disability status and MOSART posttest scores.

Within the dataset, 42 student scores were missing from the posttest, 61 student scores were missing from the pretest, one measurement was missing for whether a student was a multilingual learner, and 117 student scores were missing from the average CBM score. An imputation of the data was run using Blimp, a statistical program used to address missing data. All other analyses were conducted using the statistical package STATA-IC 17.

2.5 Treatment fidelity

The researchers monitored the teachers' implementation of taught practices in order to make sure a minimum level of performance was met. Teacher moves during the observed science lessons were recorded using the CT Scan. Prior to intervention, middle-school science teachers who received the CAP-PD intervention taught vocabulary-term meanings slightly longer (M = 134.14 s; SD = 173.56) than their colleagues who would not (M = 107.73; SD = 155.11); however, this difference was not statistically significant (F(1,64) = 0.424, p = 0.517).

Due to the violation of the homogeneity of variance (F(1,111) = 22.192, p < 0.001), a Welch ANOVA was used to analyze differences between the teachers who participated in the CAP-PD and those who did not. On average, teachers in the CAP-PD group spent 124% more time using evidence-based vocabulary practices compared to their colleagues that did not receive the CAP-PD (M = 381.99; SD = 359.20 and M = 170.79; SD = 27.71, respectively). This difference in time using the vocabulary practices was statistically significant (F(1,110.693) = 19.20, p < 0.001). On average, participants in the CAP-PD group used approximately three vocabulary strategies before receiving the professional development (range: 0–6 practices). After receiving the CAP-PD, the average number of used practices increased to about 13 (range: 8 to 18 practices). Teachers in the comparison group increased the average number of practices used from approximately four (range: 0–7 practices) in the first three observations to five for the last three observations (range: 1–7 practices). The most common practices used by the teachers included student-friendly definitions (22.15% of practices used), application of the terms (17.45%), and examples (16.11%). Two observers attended 20% of the observations for interscorer reliability purposes. An agreement of 100% on study variables of interest was achieved.

3.1 Question 1 results

The first question asked in this study was: To what extent does having a teacher participate in the CAP-PD make a difference in the post-MOSART scores of students with and without DB disabilities?

To determine whether there were significant differences in MOSART pretest scores based on DB disability status, we ran an additional ANOVA. A significant difference in MOSART pretest mean scores was yielded between students without DB disabilities whose teachers had CAP-PD access (M = 4.49, SD = 1.69) and whose teachers who would not (M = 4.87, SD = 2.02; F(1,807) = 7.97, p = 0.005); however, the difference of 0.38 points is not educationally meaningful. For students with DB disabilities, there was not a statistically significant difference between those whose teacher would have access to the CAP-PD (M = 4.14, SD = 1.50) and those whose teachers would not (M = 4.27, SD = 1.75) in MOSART preset scores (F(1,108) = 0.16, p = 0.687). For a visual of mean differences between DB disability-status subgroups, see Figure 3. See Table 4 for descriptive analysis for the different participant groups.

Two-way factorial ANOVA analyses were conducted to determine whether students with and without DB disabilities performed better on the MOSART posttest when their teacher participated in CAP-PD compared to their peers whose teacher did not participate in CAP-PD. Specifically, analyses of the between-subject effects for DB disability status, teacher participation in CAP-PD, and their interaction effects were conducted. In addition, a Levene's Test was conducted to analyze the level of homogeneity of variance. Homogeneity of variance was violated, F(3, 934) = 3.376, p = 0.018, in part due to the varying sizes of the four groups (ranging from N = 52 to N = 459), since there are fewer students with DB disabilities represented in the school system than not. As such, the F-test was not a reliable measure for a factorial ANOVA. The results for the two-way ANOVA can be seen in Table 5 and comparisons between the groups can be seen in Table 6. Since multiple comparisons were made simultaneously in the factorial ANOVA, a Bonferroni correction was used to reduce the chance of a Type 1 error. Results from the factorial ANOVA indicate that teacher participation in CAP-PD had a large effect on their students' MOSART posttest scores (d = 0.911) whereas student DB disability status had a small effect (d = 0.390) on their MOSART posttest performance. The interaction between student DB disability status and teacher participation in CAP-PD had a negligible effect on student MOSART posttest performance (d = 0.091).

To better answer Question 1, the participants were split into one of four groups based on their DB disability status and their teacher's participation in CAP-PD. A one-way ANOVA was conducted to further analyze group means and conduct post hoc testing to determine whether significant differences existed between groups. Due to the continued violation of variance (F(3,934) = 3.465, p = 0.016), a Welch ANOVA was used, since it is not sensitive to heterogeneity of variance. A significant difference in performance was present between the four groups (Welch F(3, 152.441) = 80.339, p < 0.001). The effect size of the difference in student performance on the MOSART posttest between groups whose teachers participated in CAP-PD and those who did not was large (d = 0.929).

To closer evaluate differences between groups, a Games-Howell post hoc test was conducted. The Games-Howell post hoc test is used when homogeneity of variance cannot be assumed, since it is not sensitive to differences in variance. Students without DB disabilities whose teacher participated in CAP-PD (M = 8.55, SD = 2.71) did significantly better than students with DB disabilities whose teacher participated in CAP-PD (M = 7.60, SD = 2.29, p = 0.04), students without DB disabilities whose teacher did not participate in CAP-PD (M = 6.20, SD = 2.57, p < 0.001), and students with DB disabilities whose teacher did not participate in CAP-PD (M = 4.89, SD = 2.00, p < 0.001). Similarly, students with DB disabilities whose teacher participated in CAP-PD did significantly better on the MOSART posttest compared to students with and without DB disabilities whose teacher did not participate in CAP-PD (p < 0.001 for both groups). Students without DB disabilities whose teacher did not participate in CAP-PD did significantly better on the MOSART posttest compared to students with DB disabilities whose teacher did not participate in CAP-PD (p < 0.001). See Table 5 for more information.

3.2 Question 2 results

The second question was: To what extent is student DB disability status associated with science achievement, as measured by student post-test scores holding race, gender, and pre-test scores constant? The intercept (6.04) is the grand mean of the MOSART posttest across the entire sample. On average, students without DB disabilities scored 0.99 points higher on the MOSART posttest compared to student with DB disabilities in the same classroom holding group, pretest scores, and English proficiency constant (p < 0.001). In addition, students whose teachers participated in the CAP-PD scored on average 2.39 points higher on the MOSART posttest scores compared to students whose teachers did not participate, holding DB disability status, English proficiency, and pretest scores constant (p < 0.001). A one-point increase in student MOSART pretest is associated with a 0.29 point increase on the MOSART posttest, holding DB disability status, group, and English proficiency constant (p < 0.001). English language proficiency was not significantly associated with student MOSART posttest scores. In terms of variability, about 2.6% of the variance in students' scores could be attributed to the teacher level whereas 7.0% variance in MOSART posttest scores could be accounted for based on being students in the same class with the same teacher. See Table 7 for more information.

3.3 Question 3 results

The third question was: To what extent is the interaction between DB disability status and treatment exposure associated with student post-test scores holding English proficiency and pre-test scores constant? The intercept (6.05) is the grand mean of the MOSART posttest across the entire sample. The interaction between student DB disability status and their teacher's access to CAP-PD yielded a nonsignificant association with the students' MOSART posttest scores (β = −0.66, p = 0.22). However, student DB disability status (β = −1.27, p < 0.001) and whether the students' teacher participated in CAP-PD (β = 2.39, p < 0.001) were significantly associated with student MOSART posttest scores. In other words, when controlling for the interaction between DB disability status and group membership, group alone, English proficiency, and pretest scores, participants without DB disabilities scored 1.27 points more on the MOSART posttest than students with DB disabilities. Additionally, students whose teacher participated in CAP-PD earned 2.39 points more on the MOSART posttest compared to students whose teacher did not when holding DB disability status, English proficiency, and pretest scores constant. Additionally, a one-point increase in student MOSART pretest scores is associated with a 0.29 point increase of the posttest (p < 0.001) holding DB disability status, English proficiency, and groups constant. Similar to the results from Question 1, about 2.6% of the variance in students' scores could be attributed to the teacher level whereas 7.0% variance in MOSART posttest scores could be accounted for based on being students in the same class with the same teacher. See Table 8 for more information.

3.4 Question 4 results

The fourth question was: To what extent are student average CBM scores associated with science achievement as assessed by post-MOSART assessment scores holding DB disability status, pre-MOSART assessment scores, and English proficiency constant? The intercept (6.05) is the grand mean of the MOSART posttest across the entire sample. Students' average CBM scores were significantly related to their MOSART posttest scores. Specifically, a one-point increase on students' average CBM scores is associated with 0.39 increase on students' MOSART posttest scores, holding DB disability status, English proficiency, and pretest scores constant (p < 0.001). About 3.3% of the variance in students' scores could be attributed to the teacher level whereas 9.1% variance in MOSART posttest scores could be accounted for based on being students in the same class with the same teacher. See Table 9 for more information.

4.1 Implications

The findings from this study extend upon the work of Kennedy et al. (2017, 2018) and Lauterbach et al. (2020), demonstrating the promise of providing science vocabulary professional development to secondary-school science teachers for improving student outcomes. As science teachers do not regularly provide evidence-based vocabulary instruction to their students, in part due to a lack of training to teach term meanings (Johnson & Massey, 2012), offering effective, research-driven professional development may help science teachers improve the experience of students when learning complex but needed science vocabulary. More research must be done with a larger sample of teachers and students to determine the degree to which the results could be generalized nationally.

Across all models, student English proficiency had a nonsignificant association with their MOSART posttest scores when controlling for their DB disability status, MOSART pretest scores, and whether their teacher participated in CAP-PD. This would suggest that multilingual students did not score significantly different from their peers whose first language is English on the science assessment. This might suggest that teacher participation in CAP-PD could help support multilingual students in gaining some of the needed science academic language. Although this was beyond the scope of the current study, future studies should investigate whether teacher participation in CAP-PD would make a significant difference in student achievement on the MOSART posttest for multilingual students with and without DB disabilities.

4.2 Limitations

This study is limited by the number of participating sixth-grade teachers; the sample was underpowered, making it difficult to detect group differences on the treatment variable (MOSART scores). This limitation makes the results hard to generalize to other science-education teachers who work in inclusive classrooms.

The results of this study support that indirect exposure to the CAP-PD has promise in enhancing students with and without DB disabilities' science achievement. In addition, the results reinforce the link between science vocabulary knowledge and achievement. However, the way specific teacher practices were used during these lessons were not considered beyond ensuring treatment fidelity.

As such, the question could be asked whether simply having a teacher who has access to CAP-PD is enough to improve science achievement, or if the quality and/or quantity of certain strategies used from the CAP-PD makes a difference in student outcomes. Future studies will need to investigate how the role of teacher behaviors regardless of having access to the CAP-PD impacts student science vocabulary and content knowledge growth. Specifically, does the use of a particular strategy or activity make a bigger difference in students science outcomes compared to using a different one?