Assessment of the quality, content and perceived utility of local maternity guidelines in hospitals in England implementing the saving babies’ lives care bundle to reduce stillbirth

Clinical Practice Guidelines

The Institute of Medicine defines clinical practice guidelines as ‘statements that include recommendations intended to optimise patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options’.5 Evidence-based clinical practice guidelines can improve both the process and structure of care and can improve outcomes, although a systematic review noted significant variation in effect sizes of such improvements.6 Salient examples of improved outcomes in maternity care include implementation of guidelines for antenatal care,7 diagnosis of neonatal hypoglycaemia8 or sepsis.9 Conversely, reviews of perinatal deaths have identified that national clinical guidelines were not followed in a high proportion of cases.10–12 The role of clinical practice guidelines in maternity care is strongly supported by the UK Royal College of Obstetricians and Gynaecologists (RCOG) who emphasise that ‘optimal standards of clinical care will be achieved only by following national guidelines and through the quality of staff training and clinical research.’13 Importantly, clinical practice guidelines must be effectively implemented to lead to improvements in the quality of care; however, levels of implementation vary considerably. Factors which increase the likelihood of clinicians adhering to guidance include: evidence-based nature of the recommendations, recommendations which are not controversial or in agreement with the clinicians’ views, those with clear and specific recommendations and also that the practitioner has the time and resources to perform the recommended practice.14–16 However, several studies have demonstrated that guidelines in maternity care are often of low and variable quality in these domains.17–20 Consequently, variable implementation of clinical practice guidelines may underpin variations in clinical practice which would impair uptake of quality improvement initiatives such as SBLCB. This study was undertaken to describe the variation in content and quality of clinical practice guidelines relating to the SBLCB and to report staff views and experiences of using them. It was anticipated that this information would aid ongoing implementation of the SBLCB and other quality improvement initiatives in maternity care.

Patient and public involvement

As this was an analysis of clinical practice guidelines, it was not appropriate to involve patients and the participants in this aspect of the SPiRE study.

Assessment of guideline quality using AGREE II

Maternity guidelines in relation to each element of the SBLCB were requested from all 19 trusts between July 2017 and December 2017. Participating trusts were asked to submit their current clinical guidelines relating to the four components of the SBLCB: (1) Reducing smoking in pregnancy; (2) Risk assessment and surveillance of pregnancies for FGR; (3) Raising awareness among pregnant women about detecting and reporting reduced fetal movement (RFM) and (4) Effective fetal monitoring in labour. Guidelines that were under review at the time of request were excluded from the assessment. Guidelines that were currently in active use although the expiry date had passed were reviewed in the assessment.

Guidelines were reviewed and scored by two independent observers (from YZL, SK, GLS, SR, AEPH) using the AGREE II tool which has previously been employed to assess maternity guidelines.17–20 The tool comprises 23 items categorised into six domains: Scope and Purpose, Stakeholder Involvement, Rigour of Development, Clarity of Presentation, Applicability and Editorial Independence. A quality score between 1 and 7 was generated for each of the 23 items, with 7 being the highest possible quality, expressed as a percentage. In addition, each observer was asked to provide an overall score (from 1 to 7) for each guideline and whether they would recommend use of the guideline. The mean of the reviewer’s scores for the different components were used for subsequent analysis. Item 4 (from Stakeholder Involvement domain ‘The guideline development group includes individuals from all the relevant professional groups’) and item 13 (from Rigour of Development domain ‘The guideline has been externally reviewed by experts’) were excluded from the assessment as they were not relevant to local guideline development; scores were modified accordingly to the AGREE II criteria.22 Where maternity units submitted more than one guideline for an element (eg, intrapartum monitoring), the highest score was recorded. It was anticipated from an earlier analysis that Domain 6 ‘Editorial Independence’ would not be relevant to the NHS context,20 as this addresses personal financial interests of the authors, so a sensitivity analysis excluding this domain was conducted.

Assessment of guideline content related to SBLCB

The recommendations in the unit guidelines were compared against 12 recommendations in the SBLCB (3 for element 1, 5 for element 2, 2 for element 3 and 2 for element 4, table 1). For each SBLCB recommendation, a score of 2, 1 or 0 was assigned for fully, partially or not included in the unit guideline, respectively. For each element, a score was calculated by the sum of the score for each recommendation divided by the maximum possible score for each element, expressed as a percentage.

Staff survey of views regarding clinical guidelines

The methodology was described in the study protocol21; briefly, all health professionals involved in delivering maternity care (midwives on antenatal ward, antenatal clinic, community-based staff, antenatal assessment unit and labour ward, sonographers, junior doctors and consultant obstetricians, Clinical Directors and Heads of Midwifery) were invited to participate in an online or paper survey if they had been employed in their current Trust prior to the launch of the care bundle initiative in April 2015. The survey addressed a variety of staff experiences of implementing the SBLCB and staff views about the unit culture and experiences of the use of clinical practice guidelines in their maternity unit. The responses regarding staff perception and experiences of using clinical practice guidelines were analysed and presented in this manuscript. Agreement with statements was assessed using a series of Likert scales ranging from Strongly Agree to Strongly Disagree. Surveys were carried out anonymously and were completed between July and December 2017.

Analysis

To assess inter-rater variability of AGREE II scores, Fleiss’s kappa was calculated where there were three or more appraisers per guideline. Pearson’s correlation coefficient was used to assess the relationship between the overall AGREE II score assigned by the assessor (1–7) and the total of the individual domain scores.

If completed on paper, survey data were transcribed on to the online template. Staff survey results were downloaded from the online questionnaire (Selectsurvey.net) and processed in R (www.R-project.org). Descriptive statistics were used to summarise participant responses using frequencies and percentages.

Response and characteristics of guidelines received

A total of 75 clinical practice guidelines were received from 19 trusts (figure 1). Seventeen units (89%) submitted a guideline for smoking cessation, 18 units (95%) submitted guidelines for FGR and RFM and all participating trusts returned clinical guidelines for intrapartum fetal monitoring. The majority of guidelines for FGR, RFM and intrapartum monitoring focused solely on the topic of the guideline, and this was not true for guidelines for CO monitoring and smoking cessation in pregnancy which were often contained within a generic antenatal care guideline. Only 8 trusts had stand-alone guidance about smoking cessation in pregnancy and 1 trust had a guideline specifically for ‘CO’ testing in pregnancy. Two trusts submitted more than one guideline for each element (eg, general antenatal care and smoking cessation as well as intrapartum monitoring for low-risk and high-risk pregnancies). Consequently, 72 guidelines were included in the final analysis, 6 of which (8%) were out of date and 7 (10%) had no review date (figure 1).

Guideline quality scores using AGREE II

There was significant variability in appraisers’ scores ranging from k=0.04 to k=0.39. However, the total domain scores and the independently assigned overall score were well correlated (r=0.78) indicating a good relationship between the overall score and the sum of the individual domains. For each element, the overall scores for the unit guidelines showed considerable variation (figure 2A), but the median overall scores for each element ranged from 50% (element 3) to 58% (element 1). There was no difference in the overall score between guidelines for different elements. Following review, only 4 (5.6%) guidelines were recommended for use in clinical practice without modifications and 54 (75.0%) were recommended for use subject to modifications usually making a clearer link with underpinning evidence and including an assessment of barriers/facilitators to implementation and audit standards (figure 2B). Twelve (16.7%) guidelines were not recommended for use in clinical practice for reasons such as recommended practice falling below national standards and significantly exceeding the expiry date which led to obsolete recommendations; six of these were for RFM.

Some individual domains scored higher than others (figure 2C), ‘Scope and Purpose’ (domain 1) and ‘Clarity of Presentation’ (domain 4) received the highest scores for all four guideline categories with a median score of 80% or more. This was because the objective and target population of the guideline was frequently well described and guidelines were clearly presented. In contrast, ‘Rigour of development’ was lower for all four elements categories, with most having a score of less than 50%. This was due to source references being frequently omitted making it difficult to link recommendations with the supporting evidence. Although most included a review date, the procedures for updating the guidelines were often not provided. Stakeholder involvement (domain 2) also scored low (48%) likely reflecting inadequate evidence of consultation with service users to obtain their views and preferences during guideline development. Applicability (domain 5) also had a low score (40%) with most units failing to identify barriers to implementation and resources required to implement the guideline recommendation(s). Editorial independence (domain 6) had the lowest score as few guidelines reported whether their authors had any conflict of interests. A sensitivity analysis excluding this domain made no impact to the findings (data not shown).

Unit guideline recommendations and agreement to SBLCB

The 12 recommendations from the SBLCB are shown alongside the number of guidelines that either fully or partially included each recommendation in table 1. The first element was the most fully adopted within local guidelines, with the majority of guidelines containing all three SBLCB recommendations in element 1; CO testing of all pregnant women at antenatal booking was included in 76.4% of guidelines and referral to smoking cessation services with follow-up was included in 64.7% of guidelines. The second element (FGR) was rarely fully included in unit guidelines, but all contained a form of the SBLCB recommendations for the assessment of fetal growth using serial ultrasound scanning, estimated fetal weight derived from ultrasound measurements and measurement of symphysis fundal height. The risk-assessment algorithm for FGR was present in 5% of unit guidelines. Of particular concern was that none of the guidelines contained the SBLCB recommendation for ongoing audit and reporting for small-for-gestational age babies (which also relates to the low score for applicability in the AGREE II assessment). Recommendations for information giving for RFM (element 3) were less frequently present than the proposed management algorithm (33% vs 94%). Few guidelines for intrapartum fetal monitoring (32%) mentioned the need for annual training in cardiotocography (CTG) assessment and none fully included this recommendation, whereas a buddy system for CTG interpretation was included in all guidelines to some extent.

Staff opinions on the use of clinical guidelines

One thousand and sixty-four health professionals completed the survey across the 19 maternity providers (characteristics of participants are shown in online supplementary file 1). The number of responses ranged from 17 to 126 per institution. The majority of responses (78.0%) were from midwives, although responses were received from all relevant professional groups; 69% of respondents were involved in delivery of antenatal care. The median duration in respondents’ current role was 7 years with an IQR from 3 to 12 years. The majority of staff held positive views about guidelines, stating that guidelines were important for delivering high-quality consistent care to women (table 2). However, over 30% of staff said they did not have time to use to guidelines and 24% said they were not able to implement their recommendations, which in some cases was due to a lack of equipment (eg, CO monitors). Generally most staff found their guidelines readable although 16% said they are not easily accessible although this varied from 6% to 39% between different trusts.

Key findings

This study of maternity guidelines from 19 NHS providers found that staff generally held positive views about clinical guidelines and most units had clinical guidance covering all components of the SBLCB. However, the available guidance was of variable quality and often did not contain central recommendations from the SBLCB. The variation in content and quality did not affect all aspects of clinical guidance: the ‘scope and purpose’ and ‘accessibility’ domains scored significantly higher than ‘rigour of development’ and ‘applicability’. Most clinical guidance relating to the SBLCB required some modification to be optimised for clinical use.

Strengths and limitations

This study was strengthened by the study of a large number of guidelines for different areas of maternity care provision covering aspects of both antenatal and intrapartum care covered by the SBLCB. It applied the AGREE II tool, a standardised approach for evaluating clinical guidance. The use of this objective tool was helpful although the level of agreement between assessors varied and some domains, particularly ‘editorial independence’ were arguably less relevant in a state-funded healthcare system. When the scores were adjusted for the removal of domains perceived to be less relevant this did not alter the results. The study also combined assessment of clinical guidance with data from a range of maternity healthcare professionals from the participating organisations.

It is possible that studying guidelines from maternity units which included early adopters of the SBLCB is not representative of all UK maternity units. If this potential bias altered the findings, it would have been likely to overestimate the quality of guidelines as early adopters may have more robust quality improvement and governance programmes.

Context and implications

The findings regarding the quality of clinical practice guidelines relating to the SBLCB are largely in agreement with earlier analyses of guidelines within maternity care which show wide variation in guideline quality and lower average scores for stakeholder involvement (15%–86%), rigour of development (15%–88%) and the lowest scores for applicability (0%–61%).17 18 20 23 Similar to our findings, other evaluations conducted within the UK National Health Service and Australia found editorial independence was not reported in guidelines; thus, assessment of this domain is viewed as not as informative in state-funded healthcare systems.18 20 23 A review of WHO guidelines found that newer guidelines scored more highly17; we were only able to assess changes to clinical practice guidelines over time for one element of SBLCB. There has been improvement in guideline scores for RFM since 2013 with increasing average scores in scope and purpose (71% to 92%) and clarity of presentation (80% to 88%), but there was no improvement or deterioration in lower scoring domains such as stakeholder involvement (66% to 65%), rigour of development (64% to 47%) and applicability (38% to 39%).20 Efforts to implement the SBLCB should address these domains, particularly ensuring the national recommendations from the SBLCB are contained within the unit clinical practice guidelines and a link is made between the underpinning evidence and the recommendations.

The observed variation between practices recommended in local guidelines compared with national guidance may be because the latter is often not based on high-grade evidence. Analyses demonstrate that only 9%–12% of recommendations in guidelines from the RCOG and 33% of recommendations from the American College of Obstetricians and Gynecologists are based on Grade A evidence.24 25 Furthermore, a review of 1250 recommendations in 95 guidelines of the Society of Obstetricians and Gynaecologists of Canada found that 43% of recommendations had ‘good’ underpinning evidence, but only 57% of these were obtained from at least one randomised controlled trial.26 Thus, local guideline authors may draw different conclusions from those of the national guidance and consequently not reference the underpinning evidence which may account for the low scores within the ‘Rigour of development’ domain. To address this, local guidelines should clearly cite the evidence underpinning their recommendations. Furthermore, efforts to improve the quality of evidence informing maternity care must also continue.

While the majority of respondents held positive views about the role of clinical guidelines, this study identified system factors which may prevent effective implementation, including insufficient time to implement guidance, poor readability of guidelines, respondents unable to carry out all the recommendations in the guidance due to time and resources, and inaccessibility of guidelines. These challenges are not unique to the SBLCB, and similar barriers were identified in the implementation of national maternity guidance in Australia where approximately 15% of respondents had difficulty accessing guidelines and 12% were not able to perform the guideline recommendation due to a lack of training or confidence.23 Other studies evaluating the implementation of screening for gestational diabetes and implementation of smoking cessation guidelines both highlighted a lack of capacity and time, resource and funding as significant barriers to effective uptake of guidance.27 28 In our study, the comparatively low scores in the ‘Applicability’ domain were often due to lack of evidence that barriers and facilitators to implementation had been considered and/or that audit tools were not in place to check guidance was being followed. Feedback from clinical audit is essential tool to document performance and deviations from recommended practice; a study from New Zealand significantly improved compliance with a clinical practice guideline for fetal fibronectin testing by audit and training needs identified in the audit.29 This emphasises that from the outset unit guidelines should consider practical implications of how recommendations in the SBLCB can be implemented (eg, the number of CO monitors required and referral pathways for smoking cessation services for element 1), how barriers may be overcome (eg, training to have conversations about smoking cessation) and how practice can be evaluated and fed back to clinicians.

Finally, professionals’ views about guidelines and experiences of using them may influence their uptake.15 Our finding that staff largely held positive views about clinical practice guidelines is in agreement with a qualitative study of 591 maternity healthcare professionals about safety in maternity services which suggested that local implementation of national guidelines is an effective way of working and new guidelines should be rapidly disseminated.30 When disseminating clinical guidelines, it should be recognised that practitioners occasionally hold negative views about them reducing individual practitioner’s autonomy and being unsuited to the provision of individualised patient care.15 23 Mahran et al reviewed responses from 181 obstetricians at two UK maternity units and found that respondents from secondary care held more positive views about guidelines than those from a tertiary centre. As with our study, a significant proportion of staff were unable to follow clinical guidelines due to a lack of time and resource but this study also identified disagreement with the recommended practice as reason guidelines were not followed.31 This provides further evidence of the need to ensure guidelines are up to date, contain the best evidence available and that staff training should be included as an essential component of the dissemination of evidence-based guidelines. Senior staff and ‘champions’ (front-line staff with specific additional training) have important roles in promoting practice recommended by guidelines.32