The use of network meta-analysis in updating WHO living maternal and perinatal health recommendations

The usefulness of NMA evidence rests on conceptual decisions about the choice of question and reference agent

Guideline developers must decide the most logical and useful conceptual approach to structuring the evidence for decision-makers and end users, including decisions about the PICO (population, intervention, comparison, outcomes) question(s) and relatedly the choice of reference agent. Ideally, the PICO question(s) for the recommendation aligns with the systematic review PICO question(s). NMA can make it easier to address multiple treatment comparison or even multiple PICOs, however the best question strategy may be hard to determine when a single guideline must speak to different healthcare contexts. For example, the 2018 WHO uterotonics recommendations aimed to address high-income, medium-income and low-income countries, across which uterotonics availability varies from seven in higher-resource settings to one or two in limited-resource settings. Even though we had NMA evidence and would ultimately need to compare all options with one another, in the absence of widespread access to all treatments, the appropriate conceptual starting point was not a multiple treatment comparison.4

NMA generates evidence on all possible pairwise comparisons of the included interventions, which are potentially numerous. To interpret NMA results, a ‘reference’ intervention against which all other interventions are compared is chosen. The choice of reference is often placebo or no treatment, or the most commonly used comparator treatment, that is, the best-connected intervention in the network.1 While it has an important bearing on the validity and utility of the resulting recommendation, the best choice of reference agent may not always be obvious. For the uterotonics guideline, the currently recommended treatment and best choice of reference agent was oxytocin, however placebo or no treatment was also a relevant comparator in contexts where oxytocin was not available.

A large volume of streamlined evidence is still a large volume of evidence

Effectiveness evidence generated by NMA is more coherent and streamlined than referring to multiple (potentially conflicting) systematic reviews. It also leverages indirect evidence, which can improve effect estimates. However, the GDG may still need to consider a large volume of evidence. The uterotonics question included evidence on 7 interventions plus placebo or no uterotonic treatment, and 18 outcomes (see box 1); the tocolytics question included 7 interventions plus placebo or no tocolytic treatment, and 31 outcomes (see box 2). We generated a summary of findings table for each outcome, presenting effect estimates for all interventions versus the reference. This is a lot of information for the panel to assimilate, even when synthesised and streamlined in this way.

NMA findings can be difficult for guideline panels to interpret

With multiple interventions and many outcomes comes complexity. It can be hard to discern which, if any, intervention is clinically superior, especially when results vary across outcomes.20 Panel members found it challenging to understand how much weight to place on low or very low certainty evidence, and how to make decisions where findings for different outcomes appeared contradictory. For example, in figure 2 several tocolytic interventions were likely more effective at delaying birth by 7 days compared with placebo, while there was probably no difference between the intervention and placebo groups in the mean time between therapy and birth. These differences occur because the two outcomes included data from different trials. Although these issues may be familiar to GDG members considering evidence from pairwise systematic reviews, their impact is multiplied in line with the complexity of NMA.

Treatment rankings are problematic and potentially misleading

For the 2022 tocolytic guideline, we presented the treatment rankings. Rankings are produced on a per-outcome basis, and thus one drug can have different rankings for different outcomes. For instance, the SUCRA for the tocolytic intervention nitric oxide donors ranged from 1% to 100% (see table 1).

Methodologists advise caution in interpreting rankings.2 13 A higher ranking cannot be relied on to consistently identify better treatments. Differences in rank might not be clinically significant, while the relative importance of an outcome may vary. Treatments can have a higher rank without evidence that they have a better effect,2 and rankings based on low or very low certainty evidence are not conclusive.13 While effect estimates usually include a conventional level of significance such as a p value or CI, SUCRA rankings do not (although other approaches, such as the use of median ranking, can).

NMA is not a panacea for GDG decision-making

While NMA has advantages, it is not a panacea for all the challenges the GDG faces when navigating effectiveness evidence. NMA does not automatically provide ‘the answer’ as to the clinically superior intervention. Given the significant resources invested in conducting NMA, and the promise inherent in using up-to-date methods, GDG members may find this disappointing or frustrating.

NMA provides a coherent picture of the available evidence, but the evidence may be incomplete or ‘patchy’. The indirect evidence may help address any gaps but cannot resolve them all. Also, assessors may have low confidence in the evidence that is available—for example, the tocolytics NMA identified 122 trials of 13 697 women, however, most outcomes had low or very low certainty evidence.

Starting point: conceptual approach

In the example described earlier (section Challenges of using evidence from NMA in EtD frameworks) concerning the best approach to take to the 2018 uterotonics guideline, the GDG needed to develop recommendations that addressed the variable availability of uterotonic agents across high-resource, medium-resource and low-resource settings. Given this variation, in order to be useful to policy-makers and clinicians in all contexts it was crucial that the guideline first establish to what extent each agent was better or worse than placebo or no treatment. To support the GDG in tackling this issue thoroughly and systematically, although all effectiveness evidence was drawn from a single NMA, we divided the decision-making process into two sets of PICO questions. In a first phase of GDG meetings, we asked whether each intervention improved outcomes, developing individual EtD frameworks for each intervention versus placebo or no treatment. These EtDs are available in a series of web annexes that accompany the published guideline.21–26 In addition to supporting the GDG in their decision-making, this deconstructed approach had the additional benefit of providing a clear evidence base for policy-makers in diverse global contexts where only one or two treatments are available. Second, speaking to settings where all interventions are available, we prepared a single EtD that compared all interventions to a reference (oxytocin), which is available in a further web annex27 and is also included in online supplemental appendix A. While this entailed multiple GDG meetings, it meant we could be confident in the completeness and relevance of the resultant recommendations.

Oxytocin was chosen as the reference because it was the current standard of care, and the most frequently investigated uterotonic across all outcomes. For the 2022 tocolytics update (EtD included in online supplemental appendix B), placebo or no tocolytic treatment was the chosen reference because there was no standard tocolytic treatment, tocolytics were not recommended for women at risk of imminent preterm birth, and in the NMA, placebo/no tocolytic treatment was the best-connected node across most outcomes.

Evidence foundation: NMA summary of findings tables

Appraisal of the evidence on effectiveness and safety of interventions is usually captured in GRADE evidence profiles (see box 3), and as such these tables provide the basis for all GDG decision-making on the undesirable and undesirable effects and certainty of the evidence. Evidence profiles have a standard format that is familiar to guideline decision-makers, being automatically generated using GRADEpro software.28 GRADEpro does not currently support NMA results (although one pilot project has trialled a new (not publicly released) GRADEpro module to support multiple intervention comparison19). Therefore we undertook the process manually, using Excel and Word.29 30 Since we undertook this process, further detailed guidance has been published providing practical strategies for GRADE-assessing NMA effect estimates, that seeks to reduce the significant workload involved. This approach may be especially beneficial for developers of guidelines based on NMA with a large number of interventions.31 As NMA generates three relevant effect estimates for each outcome—direct, indirect and network estimates—and the certainty of evidence may differ for each, it would be hard to include a complete breakdown of all GRADE assessments in the standard ‘evidence profile’ format. Our results tables therefore more closely resembled adapted ‘summary of findings’ tables.

While there is no established format for NMA evidence tables, guidance is available and there are examples in recent literature. Yepes-Nuñez et al have explored the optimal presentation of NMA results in summary of findings tables.18 Taking as their starting point published expert guidance on the aspects of NMA that should be included (relative and absolute effects; GRADE certainty; rank probabilities; NMA geometry), the authors develop a template intended to facilitate understanding and enhance decision-making. While the paper does not constitute official GRADE guidance, we adopted many aspects of the suggested approach.

For our two sets of WHO recommendations, we included one NMA summary of findings table for each outcome, detailing effectiveness evidence for all interventions versus a reference comparator (see boxes 1 and 2, and example figure 1). All effect estimates are detailed alongside their GRADE, anticipated absolute effects for the ‘headline’ estimate (as natural frequency) and the ranking (as SUCRA). The PICO, network diagram and SUCRA graph are also provided. The network estimate is usually the ‘headline’ result, although this is not always the case (eg, where no indirect evidence is available). Our summary of findings tables differed slightly from the table described by Yepes-Nuñez et al.18 We retained effect estimates and GRADE assessments for the indirect and direct evidence, alongside the network estimate, consistent with previous GRADE Working Group advice.29 30 In line with the source Cochrane review, we used SUCRA rather than median rank to express treatment rankings, and so included the SUCRA graph. We also included explanations of network diagrams and SUCRA graphs. Like Yepes-Nuñez et al, we listed the interventions in the same order (and not by order of rank), reflecting our caution about the reliability of rankings, a challenge discussed in more detail in later sections.

Summary of findings: collating all outcomes

The 2018 and 2022 EtD frameworks included 18 and 31 detailed summary of findings tables, respectively (see boxes 1 and 2, and figure 1). To facilitate comprehension of this large body of evidence, we included a collated summary of intervention effects in the main EtD framework. The full summary of findings tables were included as an appendix (later published in web annexes to the main guideline, available as online supplemental appendices A and B).27 32

The collated summaries comprised colour-coded tables. Figure 3 shows the summary from the 2018 uterotonics update. GDG assessments of the balance of the desirable and undesirable effects must factor in the magnitude of the effect; we modified the design for the 2022 tocolytics update (see figure 2), including anticipated absolute effects (natural frequency per 1000 women, with 95% CI) for each intervention versus the reference. The colours signal benefit (green), harm (orange/red) or no difference (grey). The shades (darker to lighter) reflect the certainty of the evidence, with a neutral yellow signalling very low certainty. The narrative interpretation of each result is included for clarity, and to increase accessibility for colour-blind readers, using language that reflects guidance published by the GRADE Working Group.33

These summaries brought together evidence on the size of the desirable and undesirable effects and the certainty of the evidence in a highly digestible format, enabling the panel to begin to make assessments of the balance of effects for all interventions. Although differing in some details, developers of other WHO guidelines have produced similar tabular collations of NMA results, and the authors have observed that this approach is ‘optimal’.17 34 This adaptable format provides an accessible overview and interpretative aid for guideline decision-makers.

Summary of findings: communicating treatment rankings

For the 2022 tocolytics recommendation update, we included treatment rankings in the summary of findings tables (figure 1) and provided a collated array in an appendix (figure 4). As noted earlier, the interpretation of rankings is fraught with difficulty. As indicated by the colour-coding and narrative interpretation in figure 4, the certainty of the evidence varied widely, and we had little confidence in some ostensibly ‘high ranking’ treatments for some outcomes (signalled by the yellow boxes). Recent GRADE guidance acknowledges these challenges, and offers an approach to drawing conclusions from NMA that takes into account primarily the effect estimates and certainty of the evidence, and secondarily the rankings, however we found this approach overly cumbersome to implement given the number of outcomes involved.20 35 The collated summary of SUCRA rankings provided another prism through which panel members could assess the limitations of the available evidence.

Making judgements

Guideline panel members study the EtD before the group meets. The highly visual collated summary supported identification of treatments that signalled potentially important benefits across priority outcomes, and red flags (eg, in side effect profiles). During meetings, the GDG used the collated summary to identify a shortlist of potential candidate treatments based on their benefit/risk profiles. The panel identified the most promising three options based on key outcomes, and ruled out others that were clearly unhelpful due to having the fewest benefits and/or worst side effects. When discussing the safety and effectiveness of the shortlisted options, the panel considered whether the differences between them were clinically meaningful. This discussion focused on the magnitude of the effect and certainty of the evidence, as well as outcome importance, and was less dependent on SUCRA rankings.

In the EtD frameworks that compared multiple treatments, we modified judgement tables for all EtD domains to include rows summarising judgements for all treatments (online supplemental appendices A and B).27 32 Final judgements depended on further discussion of all EtD domains (and not just effectiveness). For example, while the tocolytics recommendation highlights nifedipine, the accompanying remarks made by the GDG note that oxytocin receptor agonists and nitric oxide donors can be effective in prolonging pregnancy, but are not available in many countries and can be more costly.5