Article Text
PDF
Abstract
Objective To evaluate the fidelity of delivery of a nurse-led intervention to enhance physical activity in patients at risk for cardiovascular diseases, the Activate intervention, by assessing: (1) self-reported fidelity of delivery; (2) observed fidelity of delivery; (3) quality of delivery of the Activate intervention and (4) nurses’ beliefs about their capability, motivation, confidence and effectiveness towards delivering the Activate intervention, including behavioural change techniques.
Design An observational study.
Setting General practices in the Netherlands.
Participants Primary care nurses (n=20) from 16 general practices.
Primary and secondary outcome measures Nurses’ self-reported fidelity was evaluated using checklists (n=282), and the observed fidelity and quality of delivery were examined using audiorecordings of consultations of the delivery of the Activate intervention (n=42). Nurses’ beliefs towards delivering the intervention were assessed using questionnaires (n=72).
Results The self-reported fidelity was 88.1% and observed fidelity was 85.4%, representing high fidelity. The observed fidelity of applied behavioural change techniques was moderate (75.0%). The observed quality of delivery was sufficient and varied among nurses (mean 2.9; SD 4.4; range 0–4). Nurses’ beliefs about their capability, motivation, confidence and effectiveness towards delivering the intervention increased over time.
Conclusions Nurses delivered most intervention components as intended with sufficient quality. Nurses believed they were capable, motivated and confident to deliver the intervention. They believed the intervention was effective to increase patients’ physical activity level. Despite the high fidelity and moderate fidelity of applied behavioural change techniques, the varying quality of delivery within and across nurses might have diluted the effectiveness of the Activate intervention.
Trial registration number NCT02725203.
- primary care
- preventive medicine
- social medicine
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This thorough evaluation of the fidelity of delivery includes the self-reported and observed fidelity, the quality of delivery and nurses’ beliefs about delivering a behavioural intervention.
The comprehensive evaluation of the fidelity allows to assess variations in delivery, enhances accurate interpretations of trial results, increases understanding of how the intervention works and enables reproducibility.
The results of nurses’ performance need to be carefully interpreted since it partly relies on self-reported data; ideally, fidelity of delivery should mainly include observational data.
Introduction
Interventions aiming at behavioural change in patients are considered complex as they contain multiple interacting components.1 2 Evaluating the effectiveness of such interventions within randomised controlled trials (RCTs) is challenging though increasingly acknowledged.1 3 4 In addition to evaluating the effectiveness of complex interventions on results of prespecified outcomes,5 it is necessary to assess the extent to which interventions are delivered as described in the protocol (intervention fidelity).6 7 Evaluating the intervention fidelity is necessary as complex interventions are susceptible to variations in delivery2 and delivery with high fidelity is challenging to achieve.8 9 Conducting an evaluation of intervention fidelity contributes to an accurate interpretation of trial results, enhances understanding of how the intervention works, allows to identify training needs or improvements of the intervention delivery and enables reproducibility.6 10 Intervention fidelity implies ongoing assessment, monitoring and enhancement of reliability and internal validity of the intervention.6 Preferably, the evaluation of the intervention fidelity needs to include five dimensions: (1) study and intervention design; (2) provider training; (3) intervention delivery; (4) intervention receipt and (5) enactment of intervention skills.6 Accordingly, parallel to the evaluation of the effectiveness of a behavioural change intervention (the Activate intervention) in a cluster RCT comparing the Activate intervention to care as usual, we planned to evaluate the intervention fidelity. The Activate intervention is a nurse-led behavioural change intervention to enhance physical activity in primary care patients at risk for cardiovascular disease (CVD).11 In the trial, 195 patients of 31 general practices throughout the Netherlands participated. For the primary outcome, the trial findings demonstrated that patients who received the Activate intervention improved their level of physical activity, however not significant, compared with patients who received care as usual after 6 months.12
Prior conducted process evaluation of the intervention covered the fidelity dimensions study and intervention design, provider training, treatment receipt and enactment of treatment skills.11 13 14 The evaluation of the intervention design showed that the use of the Behaviour Change Wheel,15 as a theoretical framework for developing the Activate intervention and use of a cluster RCT design allowed for addressing the research questions.11 The evaluation of the provider training showed that the standardised comprehensive training programme for nurses facilitated nurses to acquire essential competences to deliver the intervention, although adopting these competences was challenging.11 13 Moreover, the evaluation of the treatment receipt and enactment showed that patients understood the intervention and used taught skills well and patients performed the intervention and taught skills in their daily lives.14
To provide insight into the actual delivered content of the intervention, the fidelity of delivery of the Activate intervention needs to be assessed, which is often considered the core dimension of fidelity.3 6 16 In addition to the fidelity of delivery, evaluating the quality of delivery is recommended as this has shown to influence the effectiveness of behavioural change interventions.17 Furthermore, the extent to which nurses are engaged to deliver the intervention according to the protocol is influenced by their beliefs of their capability, motivation, confidence and effectiveness of the intervention.18 19 Therefore, the specific objectives of this study were:
To assess the fidelity of delivery of the Activate intervention by examining nurses’ self-reported fidelity according to the content, consultation duration and dose of the intervention.
To assess the observed fidelity of delivery by examining the content and duration of nurses’ delivery of the Activate intervention.
To evaluate the quality of nurses’ delivery of the content of the Activate intervention.
To gain insight into nurses’ beliefs about their capability, motivation, confidence and effectiveness towards delivering the Activate intervention.
Methods
Study design
An observational study design was used to assess the fidelity and quality of delivery of the Activate intervention. The self-reported delivery of the content, duration and dose was examined using checklists and the observed fidelity and quality of nurses’ delivery was examined using audio-recordings of intervention consultations. Nurses’ beliefs towards delivering the intervention were assessed using questionnaires which were filled out during the intervention at multiple time points.
The intervention
Activate trial
The Activate trial have been reported in detail previously.11 In short, the Activate trial is a two-armed cluster RCT aiming at enhancing physical activity in primary care patients at risk for CVD. A total of 195 patients divided over 31 general practices (as clusters) in both urban and rural areas in the Netherlands participated in the trial, of which 15 were allocated to the intervention group (n=93 patients; n=20 primary care nurses) and 16 were allocated to the control group (n=102 patients; n=16 primary care nurses). Patients in the intervention group received the Activate intervention and patients in the control group received care as usual according to the healthcare standards. The primary outcome was change from baseline to 6 months of follow-up in the number of minutes of physical activity in the moderate to vigorous category, measured with an accelerometer (personal activity monitor; Pam AM300).20 Secondary outcomes were sedentary behaviour measured with the accelerometer, self-efficacy for physical activity, patient activation for self-management and health status measured with questionnaires. Outcome data were collected at baseline, at 3 months of follow-up and at 6 months of follow-up. Patients were recruited from March 2016 to January 2017, follow-up was completed by November 2017.
Intervention development
The intervention was developed using the Behaviour Change Wheel15 and included a comprehensive behavioural analysis of (1) what hinders and facilitates patients to increase their physical activity and (2) which behaviour is needed from nurses in order to deliver the Activate intervention adequately. The application of the Behaviour Change Wheel for patients’ behaviour resulted in a selection of 17 behavioural change techniques (BCTs), described in the BCT Taxonomy V.1.21 These 17 BCTs were integrated into four nurse-led consultations during a 3-month period.
The application of the Behaviour Change Wheel for nurses’ behaviour resulted in a selection of 21 BCTs, described in the BCT Taxonomy V.1.21 These 21 BCTs were incorporated in a standardised comprehensive training programme for nurses to equip them with the necessary competencies to deliver the intervention.
The Activate intervention
Consultations were delivered at week 1, 3, 7 and 12 in patients’ own general practice. The first consultation aimed to last for 30 min, and the following three consultations for 20 min.
The first consultation aimed to enhance patients’ awareness of their behaviour and health consequences and to discuss patients’ motivation towards increasing their physical activity. The second, third and fourth consultation aimed to discuss patients’ level of goal attainment and (re)set a personal action plan. The third and fourth consultation also focused on relapse prevention.
The training programme for nurses consisted of a 1-day skills training held by a health psychologist supplemented with instructional videos with examples of how to apply the BCTs in the consultations, a scripted handbook of the content of each of the consultations and checklists (what to do when). Furthermore, nurses received two individual coaching sessions held by a health psychologist to reinforce the trained skills in applying the BCTs. The first coaching session was held once a nurse started with delivering the intervention. The second coaching session was held once a nurse gained more experience in delivering the intervention.
Patient and public involvement
Prior to the start of the trial, all study procedures were pilot tested in a small feasibility study among two primary care nurses and one patient. Based on their perceptions towards the intervention, few small adaptations were made, for example, clear instructions for patient enrolment and the layout of patient information. The participants were satisfied with the content and burden of the intervention. The nurses indicated that the training adequately equipped them to deliver the intervention.
After the Activate trial, all participating nurses and patients in the trial received a report of the study results.
Participants
The study sample consisted of all primary care nurses (n=20) from 16 general practices situated throughout the Netherlands who participated in the Activate trial and were allocated to the intervention group.12 In total, these nurses delivered the intervention to 93 patients in 334 consultations.
Data collection
Self-reported delivery of the Activate intervention
The self-reported fidelity of delivery of the intervention was assessed by filling out checklists of the discussed content, the consultation duration and dose (n=282 consultations of 86 of a total of 93 patients). These checklists (what to do when) were developed by the research team. The intervention content was structured in terms of prescribed intervention components for each of the four consultations separately (see online supplemental file 1). Nurses were asked to rate each prescribed component as ‘discussed’ or ‘not discussed’ directly after each consultation. Nurses were also asked to administer the consultation duration in minutes and the number of consultations attended (dose).
Supplemental material
Observed delivery of the Activate intervention
To observe whether nurses delivered the intervention as intended, nurses were asked to randomly audiotape one of each of the four intervention consultations. Prior to each recording of a consultation, patients were asked to verbally consent to the recording. The audiorecordings (n=42 taped by 16 of a total of 20 nurses) were used to evaluate whether nurses delivered the prescribed subsequent intervention components and applied corresponding BCTs using a self-developed coding list. Each intervention component and BCT was rated as ‘discussed’ or ‘not discussed’ (see online supplemental file 1). The audiorecordings were used to register the duration (in minutes:seconds) of the consultations.
Quality of delivery of the Activate intervention
The audiorecordings of the consultations were used to assess the quality of delivery. The quality of nurses’ counselling was assessed using the Behaviour Change Counselling Index (BECCI)22 and an additional scoring list for communication skills that was developed for this study (see online supplemental file 2). The BECCI is a validated scale to score practitioners’ use of behavioural change counselling in consultations. The BECCI consists of 11 items, which are rated on a five-point Likert scale (0=not at all to 4=a great extent).22 Two items were excluded (‘practitioner invites the patient to talk about behavioural change’ and ‘practitioner demonstrates sensitivity to talk about other issues’) since these items were not applicable. The mean score per item indicates the extent to which nurses applied behavioural change counselling while delivering the Activate intervention.
Supplemental material
To assess the nurses’ communication skills that were conditional to deliver the intervention and not covered by the BECCI, a scoring list was developed by members of the research team (see online supplemental file 2). The scoring list was checked for face validity by all members of the research team and a health psychologist. The scoring list includes five items that cover communication skills that were integrated into the 1-day training. A five-point Likert scale was used to indicate the extent to which the trained communication skills were applied (0=not at all to 4=a great extent).
Beliefs towards delivery of the Activate intervention
To explore the nurses’ beliefs towards the delivery of the Activate intervention, their beliefs towards their capability, motivation, confidence to deliver the Activate intervention and beliefs about the effectiveness of the Activate intervention were assessed using a questionnaire. This questionnaire was developed for this study by members of the research team and checked for face validity by all members of the research team and a health psychologist. The questionnaire includes four statements about beliefs towards delivering the intervention and the specific BCTs: 1. ‘I am capable of [delivering the intervention/BCT]’; 2. ‘I am motivated to [deliver the intervention/BCT]’; 3. ‘I am confident that I can [deliver the intervention/BCT]’; and 4. ‘I am convinced [delivering the intervention/BCT] is effective to enhance physical activity’ (see online supplemental file 3). Nurses rated each statement using a seven-point Likert scale (1=completely disagree to 7=completely agree). Nurses were asked to fill out this questionnaire at four consecutive time points: at the start of the 1-day training, directly after the 1-day training, after their first individual coaching session and after they finalised the intervention (n=72).
Supplemental material
Data analysis
Self-reported delivery of the Activate intervention
The checklists of the discussed intervention content were analysed by calculating the proportion of delivered intervention components for each of the four consultations separately and the intervention as a whole. Consensus criteria were used to constitute adherence to the intervention content, in which <50% constitute low fidelity, 51%–79% moderate fidelity and 80%–100% high fidelity.7 23 The self-reported duration of the consultations was analysed using the median and range in minutes. The dose was analysed by the number and percentage of patients who attended the consultations.
Observed delivery of the Activate intervention
The audiorecordings were transcribed verbatim. Two researchers independently coded the delivered content of the intervention in each of the consultations using the coding list. After coding every four to six audiorecordings, the researchers compared their findings to ensure consistent application of the coding list. Discrepancies were resolved through discussions. The inter-rater reliability was calculated using Cohen’s kappa24 and percentage agreement.25 The observed fidelity was analysed by calculating the proportion of delivered components and BCTs for each of the four consultations separately and the intervention as a whole. Consensus criteria were used to constitute fidelity of the intervention content.7 23 The observed duration of the consultations was analysed by the median and range in minutes:seconds.
Quality of delivery of the Activate intervention
Two researchers independently scored the BECCI and the scoring list covering communication skills for each of the audiorecorded consultations. After scoring every four to six audiorecordings, the researchers compared their findings to ensure consistent application of the scoring lists. Discrepancies were resolved through discussions. The inter-rater reliability was calculated using Cohen’s kappa24 and percentage agreement.25 The BECCI score and observed communication skills were descriptively analysed using the mean score and SD per item for each of the consultations separately and the intervention as a whole.
Beliefs towards delivery of the Activate intervention
To determine whether nurses’ beliefs about their capability, motivation, confidence and effectiveness of delivering the Activate intervention and applying the BCTs changed over time, the median score and corresponding IQR per belief over time were calculated.
All analyses were performed using the SPSS V.21.
Results
All participating nurses were female (n=20), had a mean age of 46.9 years (SD 10.7) and had 6.8 years (SD 4.2) of working experience with patients at risk for CVD (table 1). The majority of nurses received additional training in coaching techniques (n=16; 80%) prior to the Activate intervention.
Characteristics of participating primary care nurses
Self-reported delivery of the Activate intervention
Seventeen nurses (85%) filled out a total of 282 (84.4%) checklists to assess the self-reported fidelity of delivery of the Activate intervention. Three nurses did not fill out the checklists without any reason. In consultation 1, the proportion of delivered components was 95.8% (89.4%–100%). The proportion of the delivered components for the second consultation was 85.3% (53.4%–98.6%), for the third consultation 81.6% (48.5%–95.6%) and for the fourth consultation 87.7% (67.2%–98.3%) (see table 2 and online supplemental file 1). Nurses’ overall self-reported fidelity to delivery of the Activate intervention was high: 88.1% (48.5%–100%) (see table 2). Overall, all components were delivered; however, nurses less frequently reported discussing the use of prompts, cues or reminders (48.5%–75.9%) (see online supplemental file 1).
Fidelity of delivery of the Activate intervention
The median self-reported duration of all consultations was 20 min (10–50 min). The median duration of the consultations aligned with the prescribed duration (consultation 1: 30 min; consultation 2, 3 and 4: 20 min). However, the duration of all consultations showed a wide range (see table 3).
Duration and dose of the delivered Activate intervention
All patients allocated to the intervention group (n=93) attended the first consultation, and 73 (78.5%) patients attended all four consultations (table 3). During the intervention, 18 patients (19.4%) discontinued the intervention due to health concerns (n=6), high burden (n=3), personal circumstances (n=3), the achievement of a satisfying level of physical activity (n=3) and other reasons (n=3).
Observed delivery of the Activate intervention
In total, 42 consultations were audiorecorded (53% of the intended 80 audio-recordings) by 16 (80%) nurses. Four nurses omitted to audio-tape their consultations as they felt uncomfortable doing this. The inter-rater reliability for coding the intervention components and BCTs was k 0.83; 95% CI (0.81 to 0.85) and 88.3% agreement; 95% CI 86.9 to 89.7 indicating almost perfect agreement. The proportion of delivered intervention components for consultation 1 was 96.0% (60.0%–100%) and for the BCTs was 86.0% (80.0%–100%). The proportion of delivered components for consultation 2 was 82.9% (40.0%–100%) and BCTs was 72.5% (0%–100%). For consultation three the proportion of delivered components was 81.0% (44.4%–100%) and BCTs was 71.0% (0%–100%) and for the last consultation 82.2% (38.5%–100%) and BCTs 71.3% (7.7%–100%) (see table 2 and online supplemental file 1).
The overall observed fidelity for the intervention components was 85.4% (81.1%–96.0%) indicating high fidelity and for the BCTs 75.0% (0%–100%) indicating moderate fidelity (table 2 and online supplemental file 1). Although the majority of intervention components and BCTs were delivered, nurses rarely discussed restructuring the physical environment (0%–7.7%), restructuring the social environment (7.7%–20.0%), the use of prompts/cues (22.2%–28.5%) and past success (0%–30.8%) (see online supplemental file 1).
The median observed duration of the consultations was 18:29 min (11:11-37:39). The median duration was lower than the prescribed duration and decreased as the intervention progresses (table 3).
Quality of delivery of the Activate intervention
The inter-rater reliability for scoring the BECCI was k 0.83; 95% CI (0.81 to 0.84) and 88.3% agreement; 95% CI (86.9 to 89.5), which indicates almost perfect agreement. The overall mean BECCI score was 2.9 (SD 4.4, range 0–4) and mean scores were similar among the four consultations (2.7–2.9) (see table 4). The highest scores were seen on the statements 8 (‘Nurse actively conveys respect for patient’s choice about behavioural change’); mean score 3.4 (3.2–3.7). Nurses scored lowest on statement 5 (‘Nurse uses summaries to bring together what the patient says about the topic’); mean score 2.1 (1.8–2.2).
Quality of delivery using the Behaviour Change Counselling Index (BECCI)
The inter-rater reliability of scoring the communication skills showed substantial agreement according to Cohen’s kappa (k 0.66; 95% CI 0.57 to 0.73) and high percentage agreement (80.3%; 95% CI 75.5 to 84.7). The overall score for nurses’ communication skills was 2.9 (SD 0.4, range 1–4). Mean scores of the items were similar among the four consultations (table 5). Nurses scored highest on statement 5 (‘Nurse focuses on patient’s behaviour and patient’s efforts to increase physical activity’); mean score 3.4 (3.0–3.6). Nurses scored lowest on statement 1 (‘Nurse asks open questions’); mean score 2.4 (2.3–2.5) and statement 4 (‘Nurse asks questions rather than giving advice or filling in for the patient’); mean score 2.5 (2.3–2.8).
Quality of delivery according to the observed communication skills
Beliefs towards delivery of the Activate intervention
All nurses filled out the first questionnaire, the second questionnaire was filled out by 15 (75%) nurses and the latter by 17 nurses (85%).
After nurses followed the 1-day training, they felt they were capable, motivated and confident to deliver the intervention. The nurses were positive about the effectiveness of the Activate intervention to improve patients’ level of physical activity. Nurses’ beliefs towards delivering the Activate intervention did not substantially change over time (table 6).
Beliefs of nurses towards delivering the Activate intervention
Generally, nurses’ beliefs about their capability (median 4–5) and confidence (median 5–6) to apply the BCTs were moderate at the start of the 1-day training and consistently improved afterwards. Nurses’ motivation to apply the BCTs and their beliefs about the effectiveness of the BCTs were considerably high and consistent over time (median 5–7) (see online supplemental file 3). Nurses’ beliefs about their capability, motivation, confidence of applying the BCTs and its effectiveness tend to slightly fluctuate over time, as scores slightly decreased after their first individual coaching session (measurement 3) and stabilised or increased after finalising the intervention (measurement 4).
Discussion
This study assessed the fidelity of delivery of the Activate intervention by primary care nurses. The self-reported and observed fidelity of delivery of the prescribed subsequent components of the intervention was high. The observed fidelity of the BCTs constituted moderate fidelity. The overall observed quality of delivery was sufficient as nurses frequently applied most communication skills and behavioural change counselling skills. Nurses felt capable, motivated and confident to deliver the intervention. They considered the intervention, including the BCTs, to be effective in improving patients’ level of physical activity. Nurses’ beliefs regarding their capability and confidence improved consistently after nurses received the training. Nurses’ beliefs about the effectiveness of the BCTs intervention and their motivation to deliver the intervention remained high over time.
The high self-reported and moderate to high observed fidelity of delivery of the Activate intervention was comparable26 and higher9 27 28 than observed in other behavioural interventions including BCTs. These differences might illustrate the inconsistency in the way behavioural change interventions are implemented.27 The high fidelity of the intervention components and moderate fidelity of the BCTs might be explained by numerous reasons. First, the results of our qualitative study about nurses’ perceptions towards intervention delivery revealed that the comprehensive training programme, including a 1-day training, training tools and individual coaching, equipped nurses in acquiring the competences to deliver the intervention and boosted their delivery.13 Given the complexity of behavioural change and nurses’ tendency to easily relapse into traditional habits, adding training tools and coaching, additionally to a 1-day training, are considered to be necessary and recommended to increase the intervention fidelity.10 Furthermore, the use of training tools that can be easily used in practice is likely to increase fidelity.19 Second, nurses were instructed to adhere to the protocol to increase fidelity. Our qualitative study revealed that they tried to adhere to the consultation structure although there were challenges in delivering the intervention according to the protocol, such as distraction by patients who initiated discussion of other topics.13 Third, the qualitative study showed that nurses were engaged to acquire skills in behavioural change support as they felt a need to improve their support. Patients’ success of the intervention strengthened their engagement towards delivering the intervention and aligned with their intrinsic drive of being a nurse.13 Furthermore, nurses’ engagement towards intervention delivery was confirmed by the results of nurses’ beliefs towards delivering the intervention. The fidelity of delivery was highest at the first consultation and the subsequent consultations had slightly lower fidelity. This variation of delivery across consultations might be explained by the fact that nurses rarely discussed restructuring the physical or social environment, the use of prompts/cues and past successes, which were included in the second, third and fourth consultation. Despite the fact that they rarely applied these components and corresponding BCTs, our results showed that nurses considered themselves as capable, motivated and confident to deliver these components and BCTs and considered them to be effective. Furthermore, nurses overestimated their delivery of these components and BCTs compared with those observed. Nurses could have reported these components as being discussed while they only discussed a small proportion or only slightly touched on these components and BCTs. More emphasis during the training and coaching on how to deliver these components and BCTs might have improved the delivery since nurses were not used to applying these BCTs in their routine practice and experienced acquiring the new skills as challenging.13 However, nurses strictly adhered to deliver the core components and BCTs of the intervention, such as goal setting, action planning, review on behavioural goal(s), feedback on behaviour and self-monitoring. Tailoring the intervention to patients’ individual circumstances is inherent to behavioural interventions and might result in applying only the prescribed core components and BCTs in one consultation and thus in not achieving 100% fidelity of delivery.7 29 Therefore, strict adherence to the core components and BCTs of the intervention could be regarded as successful delivery of the intervention.30
To optimise nurses’ performance and fidelity, nurses received two individual coaching sessions after they started the Activate intervention and delivered several second consultations (coaching session one) and several third consultations (coaching session two). Our qualitative study revealed that nurses highly valued these coaching sessions as these enhanced their perceived quality of delivery.13 The coaching primarily focused on their delivery of the core components and BCTs of the intervention, which might account for the lack of their delivery of the less essential components and BCTs.
The self-reported and observed fidelity showed some discrepancies as nurses regularly rated a lower or higher adherence than observed. Such discrepancy is commonly reported in studies as it is difficult to reflect on one’s own performance and underpins the importance of observing the fidelity of delivery.8 9 26
Interventions with higher levels of fidelity of delivery are associated with the effectiveness of such interventions.31 Given the high fidelity of the Activate intervention, it is unlikely that the effectiveness of the intervention is underestimated. However, the fidelity of delivery of the BCTs was moderate and therefore showed room for improvement. In routine care, nurses insufficiently focused on behavioural change support and rarely applied BCTs.32 Therefore, one could argue whether nurses were able to apply the BCTs correctly, despite the comprehensive training. Furthermore, while most nurses highly adhered to the protocol, their quality of delivery showed room for improvement as nurses easily tend to relapse into their own consultation style of closed questioning, giving advice and filling in for patients, and nurses’ tendency to adjust the intervention to their own beliefs and feelings of comfort.13 Mastering complex interventions, such as the Activate intervention, requires tailored training tools, regular practice opportunities and ongoing coaching.13 18 The sufficient quality of delivery might have contributed to the lack of significant improvement in patients’ level of physical activity. Using a validated comprehensive scoring list might have enhanced the assessment of the quality of delivery. However, to the best of our knowledge, such a scoring list is lacking and therefore a scoring list to measure the quality of delivery was developed.
Furthermore, the fidelity and quality of delivery varied within and across nurses, which is consistent with routine care32 and might also have influenced the real delivery of the intervention.33 Moreover, despite our expectations and efforts, nurses submitted only a low number of audio-recordings across the intervention period, which might have overestimated the fidelity of delivery as these consultations are likely to represent a ‘best-case’ scenario.27 Given all methodological factors influencing the assessment of the real fidelity of delivery, it is likely to assume that these factors diluted the effectiveness of the Activate intervention.
This study has several strengths. First, the inter-rater reliability between observers of the intervention content was almost perfect. All audiorecordings were independently coded by two researchers. One of the researchers was independent of the trial, suggesting that coding is likely to reflect actual performance without influences of knowledge related to the nurses. The assessed intervention content was highly specific due to the detailed protocol and the use of the taxonomy to code the applied BCTs,21 allowing consistent and systematic coding. These aspects suggest that the observed fidelity is reliable. Second, by definition, the fidelity of delivery refers to the extent to which the core intervention components are delivered as intended, which is distinguished from how components are delivered, such as quality of delivery.7 The addition of the quality assessment is recommended10 as this has been shown to influence the effectiveness of behavioural change interventions.17 Furthermore, the assessment of nurses’ quality of delivery and beliefs of delivery of the intervention and the BCTs deepened our understanding of how and what nurses delivered.
Some limitations need to be addressed. First, nurses reported their discussed components of each of the consultations, but some nurses did not report non-delivery of specific components by leaving these components unfilled. Afterwards, nurses confirmed these components as non-delivered. Furthermore, nurses were not required to report the applied BCTs. Requiring nurses to self-report their adherence in a detailed level of BCTs might increase the accuracy but decrease adherence to their self-reports.26 The high probability of inaccuracy and incompleteness of the self-reported data might have resulted in an overestimation of the self-reported fidelity, which we were not able to verify due to the relatively low number of audiorecordings. This suggests that fidelity of delivery should be observed rather than rely on self-reported fidelity of delivery.9 26 27
Second, the 42 analysed audiorecordings represent 12.6% of the total n=334 delivered consultations, which is lower than the 20% minimum recommended.30 Our qualitative evaluation among nurses revealed that nurses perceived recording of consultations as uncomfortable and felt being judged knowing that their performance was being analysed.13 Recording all delivered consultations might have reduced nurses’ reluctance towards recording their consultations; however, this would probably have led to non-participation of nurses and patients in the trial. By using self-reports we strived to gain good insight into the fidelity of delivery. However, the comparison between the self-reported and observed fidelity urged caution due to this low number of audiorecordings. Moreover, despite that nurses were instructed to randomly audiotape their consultations, the prudency of nurses towards audiorecording their consultations might have introduced selective inclusion of recorded consultations. Therefore, the observed audiorecordings might not reflect all consultations delivered by nurses. Furthermore, we were not able to specifically compare the self-reported versus the audiorecorded consultations as the audiorecordings were depersonalised.
Third, some nurses did not fill out the checklists or audiorecord their consultations. These nurses might have shown lower fidelity of delivery.
Fourth, consultation duration varied across consultations. We did not assess whether consultation duration was associated with the degree of fidelity. Therefore, we could not evaluate whether (in)sufficient time to deliver all subsequent components of the intervention and BCTs has contributed to the degree of fidelity of the prescribed intervention content.
In conclusion, nurses delivered the prescribed components of the Activate intervention with high fidelity and applied the BCTs with moderate fidelity. The quality of delivery was sufficient. Nurses felt capable, motivated and confident to deliver the intervention and BCTs and considered the intervention, including BCTs, to be effective in enhancing patients’ level of physical activity. Several methodological factors and nurses’ variation in complex behavioural change delivery might have diluted the quality of delivery, and therefore, might have diluted the effectiveness of the Activate intervention.
Acknowledgments
We gratefully thank the general practices, patients and nurses who participated in this study. We thank Annika Haanstra, Marjolein de Bruin-van Leersum and Daphne de Vette for their contribution to the data collection. We thank Wendela de Lange for her contribution to the data analysis.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors Conceptualisation and design: HW, CDS, JCAT and MJS. Data collection and management: HW. Data analyses: HW and MHZ. Manuscript development and revisions: all authors. All authors read and approved the final manuscript.
Funding This study was funded by the Dutch Ministry of Health, Welfare and Sports, ZonMw grant number 520001002.
Disclaimer The funder had no role in the design, collection, analysis, and interpretation of data, in the writing of the manuscript, and in the decision to submit the manuscript for publication.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval This study is nested within the Activate trial, which was approved by the Medical Ethics Research Committee of the University Medical Center Utrecht with protocol ID NL54286.041.15. Data were encoded and analysed anonymously.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available on reasonable request.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.