THE SECRET AGENT SOCIETY SOCIAL SKILLS PROGRAM FOR CHILDREN WITH HIGH-FUNCTIONING AUTISM SPECTRUM DISORDERS: A COMPARISON OF TWO SCHOOL VARIANTS
We would like to thank the Department of Education Queensland and Catholic Education Queensland for consenting to be involved in this research. We are very grateful to the individual schools, staff, and families who participated.
Abstract
School is often considered an ideal setting for child social skills training due to the opportunities it provides for skills teaching, modeling, and practice. The current study evaluated the effectiveness of two variants of the Secret Agent Society social skills program for children with high-functioning autism spectrum disorders (HFASD) in a mainstream school context. Sixty-nine students aged 7–12 took part in one of two different 10-week versions of the program (structured versus unstructured) to determine their relative effectiveness. Results suggested that both program variants led to improvements in emotion regulation abilities, social skills, and behavior at school and home, maintained at 6-week follow-up. However, the structured intervention generally led to superior treatment outcomes. These results suggest that improvements in social–emotional functioning can be achieved for students with HFASD through time-limited school-based interventions. Limitations of this study and directions for future research are discussed.
Introduction
Students with autism spectrum disorders (ASD) often face social challenges in the school environment, including problems deciphering how other people feel from nonverbal clues, difficulties initiating and maintaining conversations, and challenges interpreting the intentions of others, all of which culminate in a lack of age-appropriate friendships with peers (Attwood, 2007). Research shows that children with ASD are significantly more likely than their typically developing peers to be suspended from school (Barnhill et al., 2000), to be the targets of bullying (van Roekel, Scholte, & Didden, 2010), and to suffer depression and anxiety (Kim, Szatmari, Bryson, Streiner, & Wilson, 2000).
Promoting inclusive education for children with ASD also proves challenging for the school system itself. Problem behaviors that may be demonstrated by children with ASD such as emotional outbursts, tantrums, aggression, noncompliance, and peer exclusion create substantial difficulties for school staff (Strain, Wilson, & Dunlap, 2011). These difficulties coupled with resource and funding restrictions have led many schools to struggle to support the learning needs of students on the spectrum (Australian Advisory Board on Autism Spectrum Disorders, 2010). As a result, developing and implementing effective programs that teach children social skills, emotion regulation strategies, and problem-solving strategies that can be easily delivered in general education settings should be a high priority (Australian Advisory Board on Autism Spectrum Disorders, 2010). Despite this identified need however, implementing social skills interventions within the school setting can be challenging for school staff that are often limited by time constraints, resources, and specialist training (Bellini, Peters, Benner, & Hopf, 2007).
Very few studies have examined the effectiveness of school-based social skills training (SST) programs for students with ASD (Bellini et al., 2007). One meta-analysis containing 55 studies conducted from 1980 to 2005 found that group school-based social skills interventions were only minimally effective for children with ASD (Bellini et al., 2007). The reviewed studies involving 147 students found minimal treatment effects and minimal generalization across persons, settings, and play stimuli (Bellini et al., 2007).
More recently, Lopata et al. (2012) conducted a pilot evaluation of the feasibility and efficacy of a year-long comprehensive school-based SST intervention for students with high-functioning autism spectrum disorders (HFASDs). The intervention consisted of an initial 3-week summer program followed by a 10-month intervention comprising Skillstreaming (Goldstein, McGinnis, Sprafkin, Gershaw, & Klein, 1997) group social skills lessons three times per week, twice weekly therapeutic activities to facilitate skill generalization, a daily home-school reinforcement system for targeted skills, individual emotion recognition instruction three times per week using Baron-Cohen, Golan, Wheelwright, and Hill's (2004) Mind Reading software, and once monthly parent training sessions. Research personnel and school staff delivered the intervention, with results on parent-, child- and teacher-report measures showing children improved from pre- to postintervention in their knowledge and application of social skills, and their ability to identify emotions from faces and voice tones. However, the conclusions that can be drawn from this study are limited by the small sample size (12 students), absence of a comparison group, the lack of follow-up data, and the involvement of research staff as program delivery agents. It is questionable whether the intensity and duration of the intervention would be feasible if delivered solely by school staff.
Kasari, Rotheram-Fuller, Locke, and Gulsrud (2012) addressed many of these methodological limitations in their randomized controlled trial (n = 60) comparing the effectiveness of three different school-based interventions to a control condition in improving the social skills of children with HFASD. Interventions involved 12 training sessions over 6 weeks—one consisted of direct individual social skills instruction to students with HFASD, the second involved teaching typically developing peers how to engage and support students with social–emotional challenges, and the third was a combination of both child and peer interventions. Results showed that overall, the two interventions involving peer skills training resulted in significant improvements in children's social skills and peer acceptance based on teacher-report, peer-report, and observational assessment measures, with improvements maintained at 3-month follow-up. Treatment effects for the child intervention were smaller and more temporary. Collectively, Kasari et al.’s (2012) results suggest that brief school-based interventions can improve the social skills and inclusion of students with HFASD, although conclusions regarding program feasibility and effectiveness are restricted by the delivery of the interventions by research staff (as opposed to school personnel), and the researchers’ failure to evaluate whether treatment gains generalized beyond the school environment.
To address these gaps in the literature, the current study aimed to evaluate whether variants of a published social skills intervention for children with HFASD (the Secret Agent Society [SAS] program) could be successfully delivered by school staff within a school setting. The program features a multilevel computer game and other fun games and activities that teach children how to recognize emotions in themselves and others, express their feelings in appropriate ways, talk and play with others, solve social problems, and prevent and manage bullying. The standard SAS program (Beaumont, 2010) also includes weekly teacher tip sheets, a parent workbook, and parent information sessions to support skill generalization. Results from a clinic-based randomized controlled trial of the intervention (formerly called the Junior Detective Training Program) showed that the program led to significant improvements in children's social–emotional skills at home and school, with treatment gains maintained at 5-month follow-up (Beaumont & Sofronoff, 2008).
The current study aimed to evaluate whether less intensive, more cost-effective variants of the SAS program delivered by teachers could improve the social skills, emotion regulation skills, and behavior of students with HFASD at school and home. Condition 1 involved the delivery of a manualized 10-session variant of the full SAS program (Beaumont, 2010) with no direct parent involvement, as feedback from an independent specialist school trial of the program suggested that engaging parents can be challenging within a school delivery framework (Einfeld et al., 2014). In Condition 2, school staff used the Secret Agent Society Computer Game Pack (a computer game, selection of student visual supports and instruction manuals—Beaumont, 2009) “as they saw fit.” It was hypothesized that improvements in social skills, emotion regulation skills, and behavior would occur for students in both conditions, although Condition 1 participants would make greater gains than students in Condition 2, with improvements maintained at 6-week follow-up. Given the absence of parent involvement in either condition, it was predicted that improvements would be limited to the school environment.
Method
Participants
Child Participants
Sixty-nine students (64 boys and 5 girls) aged 7–12 participated in the study (35 in Condition 1 and 34 in Condition 2). Inclusion criteria required that children had an HFASD diagnosis (i.e., Asperger's disorder, high-functioning autism, or pervasive developmental disorder-not otherwise specified) as confirmed by a pediatrician or a clinical psychologist at the time of entering the study. Participants were also required to have an IQ score of 79 or higher on the Wechsler Abbreviated Scale of Intelligence (WASI—Wechsler, 1999), to ensure that they could understand the program content.
Participant demographics are shown in Table 1. Forty-six percent of the sample had comorbid diagnoses including attention deficit hyperactivity disorder (21.7%), anxiety (18.8%), depression (5.6%), oppositional defiant disorder (4.2%), sensory processing disorder (4.3%), and speech and language impairments (4.3%), and a significant number of children were reported to have two or more comorbid diagnoses (17.4%). Participants resided mainly in areas surrounding Brisbane, Australia. Twelve participants resided in more rural areas.
| Group | ||
|---|---|---|
| Characteristic | Condition 1 (n = 35) | Condition 2 (n = 34) |
| Mean Age | 9.82 (SD = 1.63) | 9.25 (SD = 1.48) |
| Gender | ||
| Male | 33 (94.3%) | 32 (91.2%) |
| Female | 2 (5.7%) | 3 (8.8%) |
| Diagnosis | ||
| Asperger's Disorder | 57.1% | 61.8% |
| High-Functioning Autism | 17.1% | 14.7% |
| PDD-NOS | 25.7% | 23.5% |
| Mean Intelligence Quotient (IQ) | ||
| Full-Scale IQ (overall) | 99.41 (SD = 13.36) | 109.21 (SD = 15.24) |
| Verbal IQ | 97.05 (SD = 11.48) | 107.21 (SD = 14.13) |
| Performance IQ | 104.00 (SD = 18.60) | 108.52 (SD = 17.85) |
| Currently Taking Medication | ||
| Yes | 15 (46.9%) | 18 (54.5%) |
| No | 17 (53.1%) | 15 (45.5%) |
| English First Language | ||
| Yes | 31 (96.9%) | 33 (100%) |
| No | 1 (3.1%) | 0 (0%) |
| Co-morbid Difficulties | ||
| Yes | 16 (50%) | 16 (48.5%) |
| No | 16 (50%) | 17 (51.5%) |
| Learning Difficulties | ||
| Yes | 25 (78.1%) | 19 (57.6%) |
| No | 7 (21.9%) | 14 (42.4%) |
| History of Psychological Therapy | ||
| Yes, Currently | 7 (21.9%) | 13 (39.4%) |
| Yes, in the Past | 13 (40.6%) | 13 (39.4%) |
| No | 12 (37.5%) | 7 (21.2%) |
Program Facilitators
Eighteen school staff from 17 schools participated as facilitators in the study. The facilitators came from various professions including special education teachers (38.9%), guidance officers/counsellors (33%), learning support staff (22.2%), and classroom teachers (5.6%).
Procedure
Recruitment
Ethics approval for the study was granted by the Behavioural and Social Sciences Ethical Review Committee at the University hosting the trial, in addition to Catholic Education Queensland (Brisbane and Rockhampton Dioceses) and Education Queensland.
School principals were contacted via e-mail to invite them to participate in the study. Additionally, the study was advertised at a Brisbane-based conference for School Psychologists and Guidance Counsellors. Based on school preference, capacity to fulfil the program requirements and spaces available, schools were allocated to either Condition 1 (structured intervention) or Condition 2 (unstructured intervention) prior to assessment. School staff then approached families of children who might benefit from participating in the study and provided them with a study information sheet and consent form. Decisions regarding group configurations for the intervention were left to the discretion of the program facilitators who knew the child participants well. Facilitators were encouraged to group students together who were of a similar age, ability level, and gender. Due to the small number of girls in both intervention conditions, however, some intervention groups included both male and female participants.
Assessment
Interested families were screened for eligibility based on information provided by the schools and were sent a link to an online questionnaire package prior to program commencement. Measures were completed by children, parents, and classroom teachers at the beginning and end of the program and at 6-week follow-up. Where classroom teachers were unable to complete the assessment measures, program facilitators were invited to do so. The latter applied to 15 participants (22% sample).
Social Skills Questionnaire—Parent (SSQ-P) and Teacher (SSQ-T) Versions (Spence, 1995)
These measures were used to evaluate the effectiveness of the interventions in improving children's social skills. Both contain 30 statements describing different social behaviors (e.g., “Asks other kids if he/she may join in activities.”). The respondent is asked to rate how applicable each statement is based on a child's behavior over the past 4 weeks (i.e., not true—0, sometimes true—1, or mostly true—2). A total score is determined by adding the ratings of each item, with higher scores indicating greater social competence. The measure has been used to evaluate the effectiveness of several social skills programs with this population (Beaumont & Sofronoff, 2008; Mackay, Knott, & Dunlop, 2007), with good reliability and validity. In the current study, Cronbach's alphas of .92 and .87 were obtained for the parent and teacher versions of the measure, respectively.
Emotion Regulation and Social Skills Questionnaire—Parent (ERSSQ-P) and Teacher (ERSSQ-T) Versions (Beaumont & Sofronoff, 2008)
These measures were used to examine children's competence in the specific emotion recognition, emotion regulation, and social skills targeted by the SAS program. The respondent rates how often a child engages in social behaviors (e.g., “Recognizes when other people are being sarcastic or teasing.”) on a 5-point Likert scale ranging from never (0) to always (4). Parent and teacher versions of the questionnaire contain 27 and 25 items, respectively. Both measures have been shown to have good psychometric properties (Beaumont & Sofronoff, 2008; Butterworth et al., 2014), with Cronbach's alphas of .90 (ERSSQ-P) and .94 (ERSSQ-T) in the present study.
The Spence Children's Anxiety Scale—Parent Version (SCAS-P; Spence, 1998)
The SCAS-P is a 39-item parent-report measure that examines children's anxiety levels in various situations. Parents rate how often their child worries about each situation (ranging from “0 = Never” to “3 = Always”). In the current study, the total score on the scale was used to measure overall anxiety levels, and had good internal consistency (α = .95).
James and the Maths Test and Dylan Is Being Teased (Atwood, 2004a, 2004b)
These measures examine children's knowledge of appropriate anxiety and anger management strategies. Children are individually read two separate scenarios, one in which James is anxious about a math test and another in which Dylan is coping with bullying at school. Children are asked to suggest strategies that the characters could use to cope with the situations. Each appropriate response is scored one point, with higher scores reflecting a greater knowledge of appropriate anxiety and anger management strategies. The measures have been used in previous evaluations of interventions for children with ASD (Beaumont & Sofronoff, 2008; Sofronoff, Attwood, & Hinton, 2005).
Child Adjustment and Parent Efficacy Scale-Developmental Disability—Parent (CAPES-DD-P) and Teacher (CAPES-DD-T) Versions (Morawska & Sanders, 2010)
The CAPES-DD-P is a 30-item parent-report measure designed to assess the internalizing and externalizing behaviors of children with a range of disabilities. In the current trial, parents of the participating children were asked to rate each of the 30 items in terms of how true the statement was of their child over the past 4 weeks on a scale of 0 (not at all) to 3 (very much, or most of the time). Items are summed to get an overall score on the Child Emotional and Behavioral Problems Scale.
For the purposes of this study, a teacher version of the CAPES-DD was also developed. The CAPES-DD-T adopted the same format, content, and scoring as the parent version but addressed difficulties in relation to the school rather than home context. In addition to the standard CAPES-DD behavioral items, teachers were also asked to rate their confidence in dealing with each behavior on a scale from 1 (certain I can't manage it) to 10 (certain I can manage it). High teacher confidence scores indicated greater levels of teacher efficacy.
The psychometric properties of the CAPES-DD questionnaires are currently being evaluated. For the current study, the CAPES-DD-P and CAPES-DD-T total scores were found to have good internal consistency (α = .87 and α = .90, respectively).
Interventions
Condition 1
Facilitators initially watched a three-and-a-half-hour training DVD to provide them with an overview of the program aims, structure, materials and content, as well as tips for enhancing child engagement, preventing and managing difficult behavior and strategies for promoting skill usage in daily life. Facilitators then delivered the manualized intervention, which consisted of ten 90-minute (or twenty 45-minute) group sessions over 10 weeks. Each group session was delivered to a group of approximately three students.
Compared to the published SAS intervention (Beaumont, 2010), program modifications were made for the purposes of this school trial. This intervention required fewer materials (e.g., removal of the SAS Challenger Board Game and Helpful Thought Missile game) and activity modifications made use of resources that were readily accessible in a school (e.g., balloons, balls, other games) to improve cost effectiveness. To further improve program affordability, the SAS computer game was played in sessions as opposed to between sessions (i.e., as “home missions”), resulting in only one computer game per group (rather than one per child in the published program).
The SAS parent workbook and parent information sessions were eliminated from the current program. Facilitators were initially trained via an instructional DVD and offered weekly phone/e-mail support from the second author, rather than attending the standard two-day SAS face-to-face practitioner training course and receiving a 30-minute follow-up support call.
In Condition 1, the Home-School Diary that is typically used in the SAS program to reward children's skill usage at home and school was managed by one key person (the school program facilitator), and children were introduced to the “Thought Tracker” gadget as a mindfulness-based alternative to “the Helpful Thought Missile” to cope with unhelpful thoughts. The Session 5 content was also adapted to give children more time to explore the concept of friendship before learning social interaction skills in Sessions 6–10.
To check treatment fidelity, facilitators in Condition 1 completed checklists indicating which activities they completed each session. Overall, the mean percentage completion rate of session activities for the sample was 83%, with a range between 75% and 95%. Caution is warranted in interpreting these results, however, as no independent check was conducted on the accuracy of this self-report data.
Condition 2
Condition 2 facilitators were encouraged to use the Secret Agent Society Computer Game Pack materials (Beaumont, 2009) “as they saw fit” with students, integrating these resources into the existing SST they intended to do with students over the 10-week intervention period. This implementation protocol was used to optimize the external validity of Condition 2, as it closely simulates how school staff typically use this resource. The computer game included in the pack was the same as that used in Condition 1. Facilitators in Condition 2 were asked to record weekly how much time they spent playing the computer game with students and how they chose to integrate the resources provided. Brief details are shown in Table 2.
| Characteristic | Percentage of Schools (%) |
|---|---|
| Format | |
| Individual Sessions | 50 |
| Paired Sessions | 25 |
| Group Sessions | 25 |
| Contact Frequency | |
| Weekly | 80 |
| Sporadically | 20 |
| Duration Per Session | |
| 30 minutes | 20 |
| 45 minutes | 60 |
| 60 minutes | 20 |
| Use of Additional Programs With SAS Materials | |
| Yes | 42.9 |
| No | 57.1 |
| Use of Activities to Assist With Skill Generalization | |
| Home Work Activities | 28.6 |
| Role-Play | 66.7 |
| Structured Play | 0 |
| Rewards for Practice in Session | 22.2 |
| Rewards for Practice Outside Session | 22.2 |
No formal training was provided to Condition 2 facilitators, although they were encouraged to refer to the Guide for Parents and Professionals and Computer Game Instruction Manual included in the Computer Game Pack (Beaumont, 2009). The pack contains the SAS computer game and visual supports (e.g., skill Code Cards) to facilitate children's skill generalization, although does not include the additional child and teacher resources described in Condition 1. It is yet to be empirically evaluated.
Results
A series of one-way ANOVAs indicated that the groups (Condition 1 and Condition 2) were matched on age, although Condition 2 (unstructured) participants had a significantly higher mean IQ than participants in Condition 1 (structured), F(1, 61) = 7.23, p = .009. As both groups fell within the average IQ range, however, no further adjustments were made. Chi-square analyses indicated that there was no significant difference between the groups in terms of the proportion of male and female participants, the number of children with comorbid difficulties, or the proportion of children taking medication.
For the analyses described below, one child withdrew from the study due to changing schools during the intervention period. Fourteen parents and 11 teachers failed to return postintervention questionnaires and 24 parents and 19 teachers failed to return 6-week follow-up questionnaires. Follow-up data were collected at the end of the school year when competing demands on parents’ and teachers’ time made questionnaire completion difficult.
However, a missing values analysis revealed that overall, less than 5% of data were missing. Little's Missing Completely at Random test showed that the data were missing completely at random χ2(2558) = .000, p = 1.000. Consequently, all analyses were conducted on an intention to treat basis. For any participant where postintervention or follow-up data were not available, the child's preintervention scores were assigned to both the postintervention and follow-up outcome scores, essentially assuming that the child made no improvement over time.
Intervention Effects—Pre- to Postintervention
Emotion Regulation and Social Skills
A mixed between-within subjects ANOVA was conducted to assess the impact of the two interventions on participants’ scores on the ERSSQ (parent and teacher versions) from pre- to postintervention. Based on teacher data, there was a significant main effect of time, Wilks’ lambda = .70, F(1, 66) = 28.70, p < .001, η2 = .30, which was further qualified by a significant time × group interaction, Wilks’ lambda = .89, F(1, 66) = 8.175, p = .006, η2 = .11, suggesting that the level of improvement in children's emotional regulation and social skills from pre- to postintervention at school was dependent on the condition in which they were participating. Follow-up one-way repeated measures ANOVAs with a corrected alpha of .025 indicated that there was a significant improvement in the pre- to posttreatment scores of participants in Condition 1 (structured), F(1, 66) = 33.76, p < .001, η2 = .34, but not in Condition 2 (unstructured), F(1, 66) = 3.12, p = .082 (see Table 3 for mean scores).
| Pre | Post | Follow-Up | ||||||
|---|---|---|---|---|---|---|---|---|
| Measure | Condition | N | M | SD | M | SD | M | SD |
| ERSSQ-T | 1 | 34 | 43.35 | 15.15 | 55.06a | 18.42 | 56.38Δ | 18.88 |
| 2 | 34 | 50.94 | 14.58 | 54.50 | 12.85 | 56.71Δ | 14.19 | |
| ERSSQ-P | 1 | 32 | 46.34 | 10.96 | 55.28a | 13.24 | 56.38Δ | 12.90 |
| 2 | 33 | 48.39 | 12.77 | 52.67a | 13.74 | 54.81Δ | 14.78 | |
| SSQ-T | 1 | 34 | 28.79 | 11.66 | 36.03a | 13.81 | 36.38Δ | 13.88 |
| 2 | 34 | 35.06 | 9.98 | 38.79a | 10.59 | 40.71Δ | 11.01 | |
| SSQ-P | 1 | 32 | 31.72 | 10.10 | 36.19a | 12.50 | 35.28Δ | 11.00 |
| 2 | 33 | 34.48 | 10.62 | 37.54a | 11.13 | 36.58 | 11.71 | |
| SCAS-P | 1 | 32 | 30.13 | 22.00 | 22.84a | 14.69 | 23.28Δ | 14.85 |
| 2 | 32 | 31.78 | 17.93 | 29.75 | 15.37 | 28.72 | 16.39 | |
| CAPES-DD-T | 1 | 33 | 34.60 | 11.16 | 28.48a | 13.09 | 28.42Δ | 14.02 |
| 2 | 34 | 25.76 | 11.56 | 23.76 | 10.40 | 23.82 | 10.63 | |
| CAPES-DD-P | 1 | 28 | 34.57 | 8.2 | 31.14a | 7.37 | 30.61Δ | 7.97 |
| 2 | 33 | 36.24 | 11.70 | 33.45a | 10.92 | 32.18Δ | 12.18 | |
| Teacher Efficacy | 1 | 33 | 144.54 | 31.60 | 172.63a | 18.66 | 170.83Δ | 17.66 |
| 2 | 34 | 180.24 | 21.61 | 185.00 | 20.37 | 187.59 | 17.40 | |
| DYLAN | 1 | 31 | 2.10 | 1.37 | 4.16a | 2.05 | 4.68Δ | 2.03 |
| 2 | 31 | 2.32 | 1.30 | 3.10 | 1.56 | 2.68 | 1.37 | |
| JAMES | 1 | 31 | 1.52 | 1.18 | 3.39a | 1.15 | 3.68Δ | 1.62 |
| 2 | 29 | 1.48 | 1.24 | 2.52a | 1.21 | 2.41Δ | 1.23 | |
- a Significant improvement from pre- to postintervention (p ≤ .001).
- ΔSignificant improvement from preintervention to follow-up (p < .02).
However, results from a one-way between-groups ANOVA showed that at preintervention, the two groups differed on teacher-reported emotional regulation and social skills such that the children in Condition 2 were rated to be significantly more proficient than those in Condition 1, F(1,66) = 4.43, p = .039. This suggests that children in Condition 1 made significant improvements in teacher-reported social and emotional skills such that at postintervention, the children in both groups were functioning at similar levels, t(60) = .047, p = .963.
Based on parent-report data on the ERSSQ-P, there was no significant interaction between the intervention type and time, Wilks’ lambda = .95, F(1, 63) = 3.26, p = .076, η2 = .05. There was a significant main effect for time, however, Wilks’ lambda = .71, F(1, 63) = 26.16, p < .001, η2 = .29, with both groups showing an increase in emotional regulation and social skill proficiency from pre- to postintervention. The main effect comparing the two groups was not significant, F(1, 63) = .01, p = .922, suggesting no difference in the effectiveness of the two interventions based on parent report.
The effect of the interventions on children's social skills was also analyzed using the parent and teacher versions of the SSQ. Based on teacher data (SSQ-T), there was no significant interaction between the intervention type and time, Wilks’ lambda = .97, F(1, 66) = 2.03, p = .159. There was however a significant main effect for time, Wilks’ lambda = .77, F(1, 66) = 19.93, p < .001, η2 = .23, with both groups showing a significant increase in social skill proficiency across the two time periods (see Table 3). The main effect comparing the two types of intervention was not significant, F(1, 66) = 3.18, p = .079, suggesting no difference in the effectiveness of the interventions in improving students’ social skills at school.
A similar result was found for the SSQ parent-report data with no significant interaction between the intervention type and time, Wilks’ lambda = .99, F(1, 63) = .53, p = .468, but a significant main effect for time, Wilks’ lambda = .81, F(1, 63) = 15.23, p < .001, η2 = .20. This suggests that children in both conditions showed improvements in their social skills at home.
Thematic analysis of parent and teacher qualitative program feedback indicated specific social–emotional gains in areas such as students feeling happier going to school, having greater self-esteem, being included and participating more in classroom and play activities with other students, and improved emotional awareness and emotion regulation skills.
Anxiety
A mixed between-within subjects ANOVA was conducted to assess the impact of the two interventions on participants’ scores on the SCAS-P from pre- to postintervention. Results showed a significant main effect of time, Wilks’ lambda = .84, F(1, 62) = 12.13, p = .001, η2 = .16, which was further qualified by a significant time × group interaction, Wilks’ lambda = .94, F(1, 62) = 3.86, p = .05, η2 = .06. Follow-up one-way repeated measures ANOVAs with a corrected alpha of .025 revealed that this time effect was significant for Condition 1, F(1, 62) = 14.83, p < .001, η2 = .19 but not for Condition 2, F(1, 62) = 1.15, p = .287, suggesting that only parents in Condition 1 saw significant improvements in their children's anxiety levels.
Child Adjustment and Behavior
The impact of the two interventions on participants’ scores on the CAPES-DD (parent and teacher versions) was evaluated with a mixed between-within subjects ANOVA. Results from the teacher data indicated a significant main effect of time, Wilks’ lambda = .75, F(1, 65) = 21.63, p < .001, η2 = .25, with a significant time × group interaction, Wilks’ lambda = .89, F(1, 65) = 7.63, p = .007, η2 = .11. Follow-up repeated measures ANOVAs for each condition with Bonferroni corrected alpha values revealed that this time effect was significant for Condition 1, F(1, 65) = 27.08, p < .001, η2 = .29 but not for Condition 2, F(1, 65) = 1.81, p = .183. This suggests that only the behavioral concerns of children in Condition 1 improved significantly from pre- to postintervention at school. In addition, results from a between-groups ANOVA showed that at preintervention, the two groups differed on teacher-reported behavioral difficulties such that the children in Condition 1 were rated to have significantly more behavioral concerns than those in Condition 2, F(1, 66) = 11.17, p = .001. However, at postintervention there were no significant differences between the two groups in teacher-reported behavioral concerns, F(1, 66) = 2.68, p = .106, suggesting that following the intervention, both groups were functioning at similar levels (see Table 3).
Parent-report data for the CAPES-DD showed a significant main effect for time, Wilks’ lambda = .77, F(1, 59) = 17.84, p < .001, η2 = .23, but no significant group × time interaction, Wilks’ lambda = .10, F(1, 59) = 19, p = .665. This finding indicates a decrease in behavioral difficulties for students in both conditions across the course of the SAS programs. The main effect comparing the two types of intervention was not significant, F(1, 59) = .67, p = .415, suggesting no difference in their effectiveness in reducing behavioral problems at home.
Teacher Efficacy
Based on data from the CAPES-DD-T Efficacy Scale, a significant main effect of time was found, Wilks’ lambda = .72, F(1, 55) = 21.93, p < .001, η2 = .29, which was further qualified by a time × group interaction that was trending toward significance, Wilks’ lambda = .92, F(1, 55) = 5.14, p = .027, η2 = .085. Follow-up repeated measures ANOVAs revealed that this time effect was significant for Condition 1, F(1, 55) = 23.74, p < .001, η2 = .301, but was only trending toward significance for Condition 2, F(1, 55) = 2.97, p = .09 (see Table 3 for mean scores). However, a between-groups ANOVA showed that at preintervention, the two groups significantly differed on teacher-reported self-efficacy, such that the school staff in Condition 1 rated themselves as having significantly lower self-efficacy than those in Condition 2, F(1, 55) = 18.81, p < .001. At postintervention, this finding was replicated such that although school staff in Condition 1 improved significantly over the course of the intervention, they continued to rate themselves as having significantly lower self-efficacy than those in Condition 2, F(1, 55) = 9.28, p = .004.
Children's Knowledge of Emotion-Management Strategies
A mixed between-within subjects analysis of variance was performed on participants’ scores on the James and Dylan measures (see Table 3 for mean scores). Results showed that for Dylan Is Being Teased, there was a significant main effect of time, F(1, 60) = 31.06, p <.001, η2 = .34 and a significant group × time interaction, F(1, 60) = 6.42, p = .014, η2 = .10. Repeated measures ANOVAs for each condition indicated that there was a significant improvement in the pre- to posttreatment scores of participants in Condition 1, F(1, 60) = 32.86, p <.001, η2 = .35, but not in Condition 2, F(1, 60) = 4.62, p = .036. Moreover, between-groups ANOVAs with a corrected alpha of .025 showed that although the two groups performed equivalently on the measure at pretreatment, F(1, 60) = .44, p = .509, participants in Condition 1 significantly outperformed those in Condition 2 at the completion of the intervention, F(1, 60) = 5.30, p = .025.
Analysis of James and the Maths Test data showed a similar pattern of findings, with a significant main effect of time, F(1, 58) = 52.99, p <.001, η2 = .48, and a significant group × time interaction, F(1, 58) = 4.39, p = .040, η2 = 07. However, repeated measures ANOVAs indicated that there was a significant improvement in the pre- to posttreatment scores of participants in both Condition 1, F(1, 58) = 45.46, p < .001, η2 = .44, and Condition 2, F(1, 58) = 13.00, p = .001, η2 = .18. Between-groups ANOVAs showed that although the two groups performed equivalently on James and the Maths Test at pretreatment, F(1, 58) = .01, p = .915, participants in Condition 1 performed significantly better than those in Condition 2 at the completion of the intervention, F(1, 58) = 8.16, p = .006, η2 = .12, as shown in Table 3.
Maintenance of Treatment Effects
Mixed between-within subjects ANOVAs with follow-up simple effects analyses and pairwise comparisons for each condition (repeated measures ANOVAs; corrected alpha level of .01) were performed to examine whether treatment gains were maintained at 6-week follow-up. Results showed that, for participants in both the structured and unstructured interventions, any significant improvements on outcome variables from pre- to posttreatment described above were maintained at 6-week follow-up (see Table 3 for mean scores), with the exception of Condition 2 participants’ 6-week follow-up SSQ-P scores, which were no longer significantly better than their Time 1 scores.
Exclusion of Participants Receiving Concurrent Psychological Interventions
Analyses were repeated excluding all children who participated in concurrent psychological interventions during the study (see Table 1 for details). Results for Condition 1 participants remained the same. However, any improvements previously described for Condition 2 participants on parent- and teacher-report measures of emotion regulation, social skills, and behavior only trended toward significance once children involved in concurrent psychological interventions were excluded (ps = .05–.16). The disproportionate reduction in sample size for Condition 2 (n = 21) relative to Condition 1 (n = 28) for these analyses may have contributed to a relative reduction in power to demonstrate significant treatment effects for the Condition 2 intervention, which was hypothesized to result in smaller treatment effects than Condition 1 at the outset.
Discussion
The present results provide preliminary support for the effectiveness of two adapted school-based versions of the SAS program (a structured small group intervention vs. the SAS Computer Game Pack) for students with HFASD. These findings are encouraging given Bellini et al.’s (2007) meta-analysis showing that school-based social skills interventions for students with ASD resulted in minimal treatment effects and generalization across settings. The present study also extends on recent research showing more positive intervention outcomes (e.g., Kasari et al., 2012; Lopata et al., 2012) by comparing the effectiveness of two brief interventions delivered by school staff (as opposed to research personnel), with treatment gains assessed across both school and home environments.
Contrary to Kasari et al.’s (2012) findings, the moderate to large effect sizes demonstrated in this trial (particularly for Condition 1) suggest that meaningful gains in social–emotional functioning can be achieved with direct social skills instruction to students with HFASD without a major peer training component. However, the effects of the SAS interventions on peer acceptance and friendships are yet to be explored.
As hypothesized, the structured comprehensive program format (Condition 1) appeared to result in greater improvements in children's social–emotional functioning and behavior on the majority of outcome measures compared with Condition 2, where program facilitators were advised to use the SAS Computer Game Pack “as they saw fit.” However, some of these enhanced treatment effects (e.g., on the ERSSQ-T and CAPES-DD-T) may have been due to participants in Condition 1 performing more poorly than those in Condition 2 on measures at preintervention and consequently having more room to move on the scales.
Contrary to what was hypothesized, improvements in participants’ social–emotional functioning and behavior appeared to generalize to the home environment, with improvements shown for Condition 1 participants on all parent-report measures and Condition 2 participants on the ERSSSQ-P and CAPES-DD-P. These findings are unexpected given the large body of research demonstrating the challenges that children with ASD face in generalizing skills across environments (e.g., Bellini et al., 2007) and the crucial role that parental involvement is considered to play in child therapy (e.g., Sofronoff et al., 2005). Given the absence of direct parent involvement in either intervention, it is encouraging yet surprising that observable changes were seen at home.
It is possible that delivering school-based social–emotional skills interventions affords significantly more opportunities for skills practice and staff support than community-based programs. The positive reinforcement that children receive from school staff (and potentially peers) when appropriate social–emotional skills are used may also be powerful in encouraging skill application. As students in Condition 1 seemed to generalize skills to the home environment more than those in Condition 2, it may be that one or more components in the structured program (e.g., the “home missions”) was responsible for the differential generalization effect. Caution is warranted when making conclusions about skill generalization from the parent data, however, as parents’ expectations of improvement may have positively biased their questionnaire responses. To further enhance children's skill generalization, the majority of parents and program facilitators recommended that the interventions more actively involve parents (as is the case in the standard, published SAS program).
Also of interest is improvement on the child behavior measures (CAPES-DD-T and CAPES-DD-P) given that the SAS interventions do not directly target behavioral issues. Children's improved behavior may have been due to their enhanced abilities to calm down, understand social situations, and interact with others, resulting in fewer meltdowns fuelled by feelings of anxiety, anger, confusion, and exclusion. Through the weekly Teacher Tip Sheets, school staff in Condition 1 were educated on the challenges faced by their students with ASD and may have learned new and more effective strategies to prevent and manage these difficulties. This interpretation is supported by the improvement in teacher efficacy over the course of Intervention 1 on the CAPES-DD-T.
Limitations and Future Directions
The conclusions that can be drawn from this study are limited by several methodological factors. First, maturational effects and expectations of improvement may have positively biased adults’ responses on the treatment outcome measures. Given the lifelong nature of ASD, however, spontaneous, significant improvements in students’ social–emotional functioning over a short time-period are improbable. Two independent evaluations of the SAS program (Beaumont & Sofronoff, 2008; Einfeld et al., 2014) showed that participants in treatment as usual and school curriculum as usual conditions showed no improvement on the measures used in this study over a 2- to 3-month period. Furthermore, responder bias cannot explain the positive findings on the James and the Maths Test and Dylan Is Being Teased measures. Nonetheless, it is recommended that future research include an “education as usual” control group and involve observational assessments of participants’ school-based social interactions by raters blind to their intervention status. Evaluations of the impact that improved social–emotional functioning has on children's friendships and popularity among their peers, their experiences of social exclusion and bullying, their academic performance, and their overall mental health are also needed in future trials.
Future research should address other limitations of this study including the reliance on facilitators’ self-report program fidelity data (with no independent reliability check), the failure to confirm students’ current diagnostic status using established measures (e.g., the Autism Diagnostic Interview-Revised), the large number of parents and teachers who failed to return questionnaires at postintervention and follow-up (exacerbated by the timing of data collection), the absence of long-term follow-up data, and the lack of random assignment to intervention groups. Furthermore, the conclusions that can be drawn about the effectiveness of the Computer Game Pack (Condition 2) as a stand-alone intervention tool are restricted by the heterogeneity with which school staff used this resource, with 42.9% of schools using the game and visual supports in conjunction with other social–emotional curricula (as typically happens in real life), and disproportionately more children in Condition 2 relative to Condition 1 (39% vs. 21%) participating in concurrent psychological therapy during the study. Future research is needed that more rigorously controls for how the Computer Game Pack is used, and that limits children's participation in concurrent interventions where ethically possible. The comparative effectiveness and appropriateness of the interventions for males and females (given the small number of females in the current sample), and for children at either ends of the 7- to 12-year-old age spectrum are also empirical questions worthy of further investigation. Similar to Lopata et al.’s (2012) and Kasari et al.’s (2012) evaluations, there were disproportionately more male than female participants in the present study, and the age range of participants was relatively large.
Nonetheless, the results from this study provide preliminary support for the delivery of social–emotional skills training programs to students with HFASD within schools. School-based programs would help to alleviate the cost, time, and accessibility barriers that many parents face with clinic-based interventions. Schools are also well positioned to optimally support children's social skill generalization and maintenance through school staff and peer support, creating caring, compassionate school communities.
