You are here

Article Text

Article menu
Research methods
Original research
How do network meta-analyses address intransitivity when assessing certainty of evidence: a systematic survey
  1. Ying Wang1,
  2. Ruyu Xia2,
  3. Tina Poklepovic Pericic3,
  4. Geertruida E Bekkering4,
  5. Liangying Hou5,
  6. Malgorzata M Bala6,
  7. Ya Gao5,
  8. Michael Wu7,
  9. David Gloss8,
  10. Reed Alexander Siemieniuk1,
  11. Yutong Fei2,
  12. Bram Rochwerg1,
  13. Gordon Guyatt1,
  14. Romina Brignardello-Petersen1
  1. 1Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
  2. 2Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, Beijing, People's Republic of China
  3. 3Department of Research in Biomedicine and Health, University of Split, Split, Croatia
  4. 4Academic Centre of General Practice, KU Leuven, Leuven, Belgium
  5. 5Evidence Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, People's Republic of China
  6. 6Department of Hygiene and Dietetics, Chair of Epidemiology and Preventive Medicine, Jagiellonian University Medical College, Cracow, Poland
  7. 7Michael G. DeGroote School of Medicine, McMaster University, Hamilton, Ontario, Canada
  8. 8Charleston Area Medical Center, Charleston, West Virginia, USA
  1. Correspondence to Dr Ying Wang; yingwwy{at}163.com

Abstract

Objectives To describe how systematic reviews with network meta-analyses (NMAs) that used the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) NMA approach addressed intransitivity when assessing certainty of evidence.

Design Systematic survey.

Data sources Medline, Embase and Cochrane Database of Systematic Reviews from September 2014 to October 2022.

Eligibility criteria Systematic reviews of randomised controlled trials with aggregate data NMAs that used the GRADE NMA approach for assessing certainty of evidence.

Data extraction and synthesis We documented how reviewers described methods for addressing intransitivity when assessing certainty of evidence, how often they rated down for intransitivity and their explanations for rating down.

Results Of the 268 eligible systematic reviews, 44.8% (120/268) mentioned intransitivity when describing methods for assessing the certainty of evidence. Of these, 28.3% (34/120) considered effect modifiers and from this subset, 67.6% (23/34) specified the effect modifiers; however, no systematic review noted how they chose the effect modifiers. 15.0% (18/120) mentioned looking for differences between the direct comparisons that inform the indirect estimate. No review specified a threshold for difference in effect modifiers between the direct comparisons that would lead to rating down for intransitivity. Reviewers noted rating down indirect evidence for intransitivity in 33.1% of systematic reviews, and noted intransitivity for network estimates in 23.0% of reviews. Authors provided an explanation for rating down for intransitivity in 59.6% (31/52) of the cases in which they rated down. Of the 31 in which they provided an explanation, 74.2% (23/31) noted they detected differences in effect modifiers and 67.7% (21/31) specified in what effect modifiers they detected differences.

Conclusions A third of systematic reviews with NMAs using the GRADE approach rated down for intransitivity. Limitations in reporting of methods to address intransitivity proved considerable. Whether the problem is that reviewers neglected to address rating down for transitivity at all, or whether they did consider but not report, is not clear. At minimum systematic reviews with NMAs need to improve their reporting practices regarding intransitivity; it may well be that they need to improve their practice in transitivity assessment. How to best address intransitivity may remain unclear for many reviewers thus additional GRADE guidance providing practical instructions for addressing intransitivity may be desirable.

  • Systematic Review
  • STATISTICS & RESEARCH METHODS
  • EPIDEMIOLOGY

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information. Not applicable.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2023-075212

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    STRENGTHS AND LIMITATIONS OF THIS STUDY

    • We conducted a comprehensive systematic search of systematic reviews and network meta-analyses (NMAs) that used the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) NMAs approach.

    • We conducted a detailed accounting of how the systematic reviews addressed intransitivity as one domain in the GRADE NMA approach, including description of methods for addressing intransitivity, and frequency and explanation for rating down for intransitivity.

    • Although all the included systematic reviews meet our criteria to be identified as using the GRADE NMA approach, we cannot rule out the possibility that some actually used the Confidence in Network Meta-Analysis (CINeMA)—another approach for assessing certainty of evidence from NMAs.

    Introduction

    Network meta-analysis (NMA), an analytic approach used to evaluate the comparative effects of multiple alternative interventions by combining both direct and indirect evidence, has gained considerable popularity.1–3

    Using direct comparisons of intervention A versus C and B versus C as an example, NMAs can inform the relative effect of intervention A versus B through indirect comparisons via the common comparator C. One core assumption underlying the indirect comparison is transitivity: that is, effect modifiers between the direct comparisons A versus C and B versus C are similar enough to justify pooling.1 4

    The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group has developed an approach for assessing the certainty of evidence from NMAs which involves first rating the direct and indirect evidence separately.5 6 Intransitivity represents one of the reasons for rating down the certainty of indirect evidence, and thus, when combined with the direct evidence, may influence the certainty of the network estimate. The GRADE NMA approach suggests that if effect modifiers differ in important ways across the direct comparisons that contribute to a specified indirect estimate (such as trials of A vs C and of B vs C that form the indirect comparison of A vs B), the likelihood of intransitivity may be high and therefore one should consider rating down the certainty of evidence from the indirect estimate.5 6

    Although this conceptual basis of considering intransitivity as one reason for rating down the certainty of indirect evidence in the GRADE NMA approach is clear,7 8 the practical application of this domain can be challenging. The challenges include how to identify important effect modifiers, how to compare effect modifiers between direct comparisons that form the basis for indirect comparison with different levels of effect modifiers, and deciding on a threshold for considering that these effect modifiers are different enough to warrant rating down for intransitivity. This remains relatively underdeveloped in terms of guidance from the GRADE working group.

    To understand how NMA authors are addressing intransitivity (whether they addressed intransitivity and considered effect modifiers as suggested by the GRADE NMA approach, and how they solved problems in practical applications of the intransitivity domain), we conducted a systematic survey of systematic reviews (SRs) with NMAs in which authors used the GRADE NMA approach for assessing certainty of evidence. We determined how often authors rated down for intransitivity and how they described this assessment in the methods and results of their SRs.

    Methods

    Eligibility criteria

    We included SRs of randomised controlled trials in which authors conducted aggregate data NMAs and used the GRADE NMA approach for assessing the certainty of evidence.5 6 To be identified as using the GRADE NMA approach, SRs must have met at least one of the following criteria:

    1. cited at least one of the GRADE NMA approach general guidance papers5 6 when describing their assessment of certainty of evidence in the Methods section of the article;

    2. reported that they used the GRADE approach, and had at least one sentence describing an assessment of certainty of evidence for indirect or network estimates consistent with the GRADE NMA approach;

    3. reported explicitly that they used the GRADE NMA approach or the GRADE approach for NMA, and absence of evidence suggesting they actually used the Confidence in Network Meta-Analysis (CINeMA) approach,9 10 an alternative method for assessing the certainty of evidence from NMA.

    We did not apply any restrictions on the topic of SRs. We excluded SRs not published in English and conference abstracts.

    Search strategy and study selection

    We searched Medline, Embase and the Cochrane Database of Systematic Reviews, from September 2014 to October 2022 (given the first GRADE NMA guidance article was published in September 2014) using a search strategy developed in collaboration with an experienced research librarian (online supplemental appendix 1). Pairs of reviewers screened, independently and in duplicate, titles and abstracts followed by full texts. Reviewers resolved conflicts by discussion or by consultation with a third reviewer.

    Data extraction

    Pairs of reviewers independently extracted the following information from each included SR, including appendices. They resolved discrepancies by discussion or by involving a third reviewer.

    Methods describing the assessment of intransitivity

    1. Whether authors mentioned the word intransitivity or not when describing methods for assessing the certainty of evidence from NMAs.

    2. For SRs that mentioned intransitivity, whether or not authors noted the following as methods for addressing intransitivity (from the least to the most detailed description):

      • Consideration of effect modifiers (ie, mention of the words ‘effect modifiers’ or ‘effect modification’, or specification of effect modifiers).

        • Specification of what factors they considered as effect modifiers.

          • Methods for identifying effect modifiers.

    3. For SRs that mentioned intransitivity, how authors expressed their evaluation of intransitivity (following are possible expressions):

      • Looking for differences between the direct comparisons that inform the specific indirect estimate (the optimal description).

      • Looking for differences across comparisons (inaccurate description).

      • Looking for differences between trials (inaccurate description).

      • Only mention of the word ‘differences’ without any other details (insufficient description).

      • No mention.

    4. For SRs that mentioned intransitivity, considered effect modifiers, and mentioned looking for differences: whether authors reported the degree of difference in effect modifiers between the direct comparisons that would lead them to rate down for intransitivity (ie, the threshold).

    Frequency for rating down for intransitivity for indirect estimates

    1. Whether authors presented certainty of evidence and reasons for rating down indirect estimates.

    2. For SRs that presented certainty of evidence and reasons for rating down, whether authors rated down the certainty of any indirect estimate due to intransitivity.

    3. For SRs that rated down for intransitivity, the total number of indirect estimates, and the number of indirect estimates rated down for intransitivity.

    4. For comparisons in which authors rated down the indirect estimates for intransitivity, whether the decision regarding rating down for intransitivity affected the final certainty rating of indirect estimates. This is relevant because rating down for intransitivity will not always result in rating down the certainty of the indirect estimate (ie, if the certainty of the indirect evidence was very low before addressing intransitivity, the judgement of intransitivity will not impact the final rating of certainty of evidence).

    Influence of rating down for intransitivity on network estimates certainty ratings

    1. Whether authors presented certainty and reasons for rating down network estimates.

    2. For SRs that presented certainty of evidence and reasons, whether authors noted intransitivity for any network estimate (ie, authors rated down the certainty of any indirect estimate due to intransitivity and the indirect estimate dominates the network estimate).

    Explanations for rating down for intransitivity

    For SRs that rated down the certainty of any indirect estimate for intransitivity:

    1. Whether authors provided an explanation for rating down, in addition to mentioning ‘serious’ or ‘very serious’ intransitivity.

    2. Whether authors noted each of the following as explanation for rating down:

      • They detected differences in effect modifiers.

        • Specified in what effect modifiers they detected differences.

          • How they chose the effect modifiers.

    3. Whether authors specified they detected differences between the direct comparisons that inform the indirect estimate.

    Rating down for indirectness

    1. For SRs that did not rate down for intransitivity, whether they rated down the certainty of all indirect estimates due to indirectness.

    2. For SRs that did not rate down for intransitivity, whether they rated down the certainty of indirect estimates due to indirectness for all comparisons that did not have direct evidence.

    Data synthesis and analysis

    We calculated the proportion of SRs in which authors rated down for intransitivity for any indirect and network estimate (ie, number of SRs that rated down for intransitivity/number of SRs that presented certainty of evidence and reasons).

    For SRs rating down for intransitivity for indirect estimates, we calculated the proportion of indirect estimates authors rated down for intransitivity (ie, number of indirect estimates that rated down for intransitivity/total number of indirect estimates), and present the median and IQR across SRs.

    For SRs rating down for intransitivity for indirect estimates, we calculated the proportion of indirect estimates for which rating down for intransitivity affected the final certainty of indirect estimates (ie, number of indirect estimates for which rating down for intransitivity affected the final certainty/number of indirect estimates that rated down for intransitivity), and present the median and IQR across SRs.

    We conducted simple logistic regression analyses to explore, at the SR level, whether the outcome rating down for intransitivity (rating down at least one comparison vs not rating down any comparison) was related to the methods that authors described for addressing intransitivity. We separately considered whether there was: mention of intransitivity at least once in the manuscript vs no mention at all, consideration of effect modifiers (ie, mention of the words ‘effect modifiers’ or ‘effect modification’ or specification of effect modifiers, vs no such mention), specification of what factors the authors considered as effect modifiers versus no such specification, specification of looking for differences between the direct comparisons that inform the indirect estimate versus no mention of looking for such differences. Our hypothesis was that one reason that some SRs did not rate down for intransitivity might be that authors did not check intransitivity sufficiently (rather than they checked and concluded problematic transitivity was absent). We performed regression analyses using Stata V.15.1.

    To explore the relationship between publication year and rating down for intransitivity, we compared the publication years between the SRs that rated down for intransitivity for at least one comparison and the SRs that did not rate down for intransitivity for any comparison.

    Patient and public involvement

    This survey did not involve patient or public in the design, or conduct, or reporting, or dissemination plans.

    Results

    Search results

    The search identified 3430 records, from which we screened 2077 titles and abstracts and 995 full texts. We included 268 SRs that used the GRADE NMA approach for assessing the certainty of evidence from NMAs (figure 1). Online supplemental appendix 2 contains a list of included SRs and a list of excluded studies.

    Figure 1
    Figure 1

    Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram of studies included in this systematic survey. NMA, network meta-analyses; GRADE, Grading of Recommendations Assessment, Development, and Evaluation; RCTs, randomised controlled trials.

    Methods for addressing intransitivity when assessing the certainty of evidence from NMAs

    Of the eligible SRs, 44.8% (120/268) mentioned intransitivity when describing methods for assessing the certainty of evidence, among which 43.3% (52/120) provided no description at all about the methods for addressing intransitivity (table 1).

    View this table:
    Table 1

    Summary of how systematic reviews described methods for addressing intransitivity

    Among the 120 that mentioned intransitivity, 28.3% (34/120)11–43 noted that they considered effect modification as the relevant issue when addressing intransitivity. Of these 34, 67.6% (23/34)11–13 15 17 19–21 23–29 32 34–37 39 40 44 further specified which factors they considered as effect modifiers, none of which specified how they identified effect modifiers. The number of effect modifiers listed ranged from 1 to 9 (median 3). Effect modifiers included population characteristics (eg, age), disease status (eg, disease severity), intervention characteristics (eg, dosage), outcome characteristics (eg, follow-up time of outcome measurement), cointerventions, trial characteristics (eg, year of publication) and trial methodology (eg, risk of bias) (online supplemental appendix 3).

    Of the 120 SRs that mentioned intransitivity, 15.0% (18/120)14–17 21 30–32 37 45–51 explicitly said they looked for differences between the direct comparisons that inform the indirect estimate, of which 215 17 specified doing this in the comparisons forming a first order loop. Although about a third of SRs mentioned looking for differences, their description was insufficient or inaccurate: 8.3% (10/120) mentioned looking for differences across comparisons, 23.3% (28/120) mentioned looking for differences between trials, and 2.5% (3/120) only mentioned the word differences without any other details.

    No SR explicitly reported how much difference in effect modifiers between the direct comparisons would warrant rating down for intransitivity (ie, the threshold). Two SRs13 19 included tables that presented mean, median or mode of effect modifiers for each head-to-head comparison; however, it is unclear how these were calculated and how authors used them to inform rating down for intransitivity. Five SRs,15 26 34 40 44 to inform whether to rate down for intransitivity, explored the impact of effect modifiers on effect estimates by conducting network meta-regression analyses. Two SRs explored the impact of effect modifiers by conducting sensitivity analysis.52 53 Two SRs54 55 addressed intransitivity by assessing whether all the direct comparisons contributing to the indirect estimate were consistent with the PICO (Population, Intervention, Comparator, and Outcome) question.

    Frequency and reasons for rating down for intransitivity

    One hundred and eighteen SRs presented assessments of certainty of indirect estimates and reasons for rating down, of which 33.1% (39/118) rated down the certainty of indirect estimates due to intransitivity (table 2). One hundred and ninety-six SRs presented certainty of network estimates and reasons for rating down, of which 23.0% (45/196) noted intransitivity for network estimates (ie, authors rated down the certainty of indirect estimates due to intransitivity and the indirect estimates dominate the network estimates). In total, 52 SRs rated down indirect estimates for intransitivity (including the 39 SRs noted rating down indirect estimates for intransitivity, and 13 SRs that did not noted rating down for intransitivity for indirect estimates but noted for network estimates).

    View this table:
    Table 2

    Frequency of rating down for intransitivity

    For the 39 SRs that rated down the certainty of indirect estimates due to intransitivity, the proportion of indirect estimates that rated down for intransitivity among all indirect estimates within one SR varies with median proportion of 32.9% (IQR 21.5%–89.7%).

    Among the 39 SRs rating down indirect estimates due to intransitivity, 15.4% (6/39)14 19 20 56–59 rated down the certainty of all indirect estimates due to intransitivity while 7.7% (3/39)22 60 61 rated down the certainty of indirect estimates due to intransitivity for all comparisons that did not have direct evidence.

    Although they did not rate down for intransitivity, 10 SRs44 58 62–69 rated down the certainty of indirect estimates due to indirectness, of which 458 66 67 69 rated down the certainty of all indirect estimates due to indirectness and other 644 62–65 68 rated down the certainty of indirect estimates due to indirectness for all comparisons that did not have direct evidence.

    Explanations for rating down for intransitivity

    Among the 52 SRs that rated down an indirect estimate due to intransitivity, 59.6% (31/52) 14 17–20 28 32 34 35 37 50 57–59 70–85 provided an explanation (table 3, online supplemental appendix 4).

    View this table:
    Table 3

    Explanation for rating down for intransitivity

    Among the 31 SRs providing an explanation for rating down due to intransitivity, 74.2% (23/31)14 17–20 28 32 34 35 37 50 57–59 70 71 76–82 noted they detected differences in effect modifiers; of which 23 SRs, 91.3% (21/23)17 19 20 28 32 34 35 37 50 57–59 70 71 76–82 specified further in what effect modifiers they detected differences. Three SRs11 76 80 mentioned how they chose the effect modifiers: 211 80 were based on prior SR results and 176 was based on prior knowledge (Appendix 4).

    19.4% (6/31) of the SRs17 18 32 57 71 76 reported that they detected differences between the direct comparisons that informed the specific indirect estimate.

    Ten SRs56 59 61 77–79 82 83 85 86 considered indirectness and intransitivity as one domain or mentioned indirectness when they reported explanations for rating down for intransitivity.

    Relationship between rating down for intransitivity and the methods they described for addressing intransitivity

    Regression analyses suggested that mention of intransitivity, consideration of effect modifiers, specification of effect modifiers and mention of looking for differences between the direct comparisons that inform the indirect estimate in the Methods section, are associated with rating down for intransitivity (table 4).

    View this table:
    Table 4

    Relationship between rating down for intransitivity and the methods they described for addressing intransitivity

    Relationship between rating down for intransitivity and the publication year

    From 2015 to 2018, the proportion of SRs that rated down for intransitivity for at least one comparison increased from 20.0% to 30.0%. After 2018, the proportion decreased (online supplemental appendix 5).

    Discussion

    Over half of the SRs using the GRADE NMA approach failed to mention intransitivity when describing their use of the GRADE methodology. Of those that did mention intransitivity when introducing methods for assessing certainty of evidence, only approximately half described their methods for addressing intransitivity. Approximately one-third of the SRs rated down the certainty of indirect estimates for intransitivity; however, only a half of them provided an explanation for rating down.

    Methodologists generally agree that judgement regarding intransitivity depends on inferences regarding effect modification.5 8 About one-third of the SRs that mentioned of intransitivity noted consideration of effect modifiers when describing methods for addressing intransitivity. For the SRs that rated down the certainty of evidence for intransitivity, most noted they detected differences in effect modifiers and specified what effect modifiers they detected differences in as explanation for rating down.

    However, authors seldom reported how they identified the effect modifiers. Two studies11 80 cited prior SRs as their rationale for choosing the effect modifiers. Indeed, identification of effect modifiers should involve not only searching for evidence of effect modification, but also assessing the credibility of effect modification.87

    To judge whether to rate down for intransitivity, one needs to judge if the effect modifiers differ in important ways between the direct comparisons that contribute to the indirect estimate.8 Our results showed that only a small proportion of the SRs clearly specified that they looked for or detected differences between the direct comparisons that inform the indirect estimate.

    Inaccurate descriptions included looking for differences across comparisons or between trials. For instance, in one NMA, there may be important differences between the direct comparisons of A versus C and B versus C, but no important differences between the direct comparisons of A versus C and D versus C. Thus, judging intransitivity for assessing certainty of evidence should be comparison specific.

    How to compare effect modifiers between the direct comparisons that form the basis for indirect comparison with different levels of effect modifiers presents challenges in practice. Two studies13 19 included tables that presented mean, median or mode of effect modifiers for each direct comparison, which may imply that they considered trials comparing the same interventions as a whole when comparing the distribution of effect modifiers. Other SRs did not make this point clear.

    Judgements regarding whether to rate down for intransitivity is a threshold issue, which involves both how much credibility of effect modification would lead to consider the effect modifiers when judging intransitivity, and how much difference in effect modifiers between the direct comparisons would lead to rate down for intransitivity. No study clearly specified or addressed either of these issues.

    Among the SRs rating down for intransitivity, proportions of indirect comparisons that rated down for intransitivity differ across studies. Notably, six studies14 19 20 56–59 rated down the certainty of all indirect estimates due to intransitivity, and three22 60 61 rated down the certainty of indirect estimates due to intransitivity for all comparisons that did not have direct evidence. These studies take the extreme position that there is always residual effect modification.

    Some SRs confused intransitivity with indirectness. Two SRs54 55 addressed intransitivity by assessing whether all the direct comparisons contributing to the indirect estimate were directly consistent with the PICO question, which is indirectness88 rather than intransitivity in the GRADE system. Ten SRs56 59 61 77–79 82 83 85 86 regarded indirectness and intransitivity as one domain or mentioned indirectness when they reported explanation for rating down for intransitivity. Although the first version of the GRADE NMA guidance6 stated indirectness refers to two concepts: (1) differences between the question of interest and the body of evidence used to inform the question and (2) intransitivity; the latest GRADE NMA guidance5 had specified the word indirectness refers only to the first concept and endorsed intransitivity as a separate issue. This inconsistency in GRADE guidance could contribute to confusion on the part of NMA authors.

    Although this survey focused only on SRs using the GRADE NMA approach, CINeMA provides another approach for assessing certainty of evidence from NMAs.9 These two approaches address intransitivity differently. The CINeMA approach addresses intransitivity in the indirectness domain.9 However, in the GRADE NMA approach, intransitivity represents a separate domain.5 6 This might explain why some SRs using the GRADE NMA approach confused intransitivity with indirectness.

    Another domain in the GRADE NMA approach that is easily confused with intransitivity is incoherence, which addresses the inconsistency between the direct and indirect estimates.5 89

    Regression analyses suggested that the SRs reporting methods for addressing intransitivity are associated with higher rate of rating down for intransitivity. This suggests that the reason why some SRs did not rate down for intransitivity might not be that they checked and failed to find intransitivity, but rather that the authors did not check for intransitivity at all.

    This is the first systematic survey investigating how the SRs that used the GRADE NMA approach for assessing certainty of evidence from NMAs described methods for addressing intransitivity, frequency of rating down for intransitivity, explanation for rating down and whether rating down for intransitivity had an important influence on ultimate certainty. Strengths of this survey include comprehensive search, transparent eligibility criteria and thorough extraction of information regarding intransitivity. We identified existing reporting problem about intransitivity, providing areas in which reviewers can improve their reporting practices. In addition, we identified the practical problems of addressing intransitivity that reviewers face—these are the issues that the GRADE working group needs to address.

    This systematic survey has limitations. Some SRs did not clearly present reasons for rating down the certainty of indirect estimates, so we cannot identify whether they have rated down for intransitivity. In accordance with the GRADE NMA approach, consideration of imprecision is unnecessary when assessing the direct and indirect estimates to inform the assessment of network estimates5; thus, some SRs did not assess imprecision for indirect estimates—we cannot determine whether their rating down for intransitivity affected the ultimate certainty of these indirect estimates. Although all the included SRs meet our criteria to be identified as using the GRADE NMA approach, because authors were sometimes unclear regarding the difference between the GRADE NMA approach and the CINeMA approach—especially for those SRs that considered intransitivity and indirectness together—we cannot rule out the possibility that some actually used the CINeMA approach. Because this is a methodological study (systematic survey) and does not meet the eligibility criteria for publication in PROSPERO,90 we did not register a protocol. We did, however, work with a protocol that we refined as we became familiar with the eligible studies. Another limitation is we did not document number of outcomes and indirect estimates for each included SR.

    In conclusion, this systematic survey showed that the majority of NMAs using the GRADE approach did not mention intransitivity as an issue in rating the certainty of indirect evidence and those that did had major limitations in reporting their methods for addressing intransitivity. Even for SRs rating down for intransitivity, reporting of the explanation for rating down proved to be often limited. Reasons why these reporting problems exist might be explained by conducting a qualitative study including interviews with authors of NMAs. However, the reporting problems may reflect failure to appropriately address transitivity or simply a reporting issue. If the problem is failure to address transitivity, additional GRADE guidance on how to address intransitivity practically may be desirable.

    Data availability statement

    All data relevant to the study are included in the article or uploaded as supplementary information. Not applicable.

    Ethics statements

    Patient consent for publication

    Ethics approval

    Not applicable.

    Acknowledgments

    We thank our librarian Rachel Couban for helping develop the search strategy and conduct the literature search.

    References

    1. ↵
      1. Caldwell DM
      . An overview of conducting systematic reviews with network meta-analysis. Syst Rev 2014;3:109. doi:10.1186/2046-4053-3-109
    2. ↵
      1. Li T,
      2. Puhan MA,
      3. Vedula SS, et al
      . Network meta-analysis-highly attractive but more methodological research is needed. BMC Med 2011;9:79. doi:10.1186/1741-7015-9-79
    3. ↵
      1. Lu G,
      2. Ades AE
      . Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med 2004;23:3105–24. doi:10.1002/sim.1875
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Baker SG,
      2. Kramer BS
      . The Transitive fallacy for randomized trials: if A Bests B and B Bests C in separate trials, is A better than C BMC Med Res Methodol 2002;2:13. doi:10.1186/1471-2288-2-13
    5. ↵
      1. Brignardello-Petersen R,
      2. Bonner A,
      3. Alexander PE, et al
      . Advances in the GRADE approach to rate the certainty in estimates from a network meta-analysis. J Clin Epidemiol 2018;93:36–44. doi:10.1016/j.jclinepi.2017.10.005
      OpenUrlCrossRefPubMed
    6. ↵
      1. Puhan MA,
      2. Schünemann HJ,
      3. Murad MH, et al
      . A GRADE working group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ 2014;349:g5630. doi:10.1136/bmj.g5630
    7. ↵
      1. Donegan S,
      2. Williamson P,
      3. D’Alessandro U, et al
      . Assessing key assumptions of network meta-analysis: a review of methods. Res Synth Methods 2013;4:291–323. doi:10.1002/jrsm.1085
      OpenUrl
    8. ↵
      1. Jansen JP,
      2. Naci H
      . Is network meta-analysis as valid as standard Pairwise meta-analysis? it all depends on the distribution of effect modifiers. BMC Med 2013;11:159. doi:10.1186/1741-7015-11-159
    9. ↵
      1. Nikolakopoulou A,
      2. Higgins JPT,
      3. Papakonstantinou T, et al
      . Cinema: an approach for assessing confidence in the results of a network meta-analysis. PLoS Med 2020;17:e1003082. doi:10.1371/journal.pmed.1003082
    10. ↵
      1. Salanti G,
      2. Del Giovane C,
      3. Chaimani A, et al
      . Evaluating the quality of evidence from a network meta-analysis. PLoS One 2014;9:e99682. doi:10.1371/journal.pone.0099682
    11. ↵
      1. Bonner A,
      2. Alexander PE,
      3. Brignardello-Petersen R, et al
      . Applying GRADE to a network meta-analysis of antidepressants led to more conservative conclusions. J Clin Epidemiol 2018;102:87–98. doi:10.1016/j.jclinepi.2018.05.009
      OpenUrlPubMed
    12. ↵
      1. Ciapponi A,
      2. Klein K,
      3. Colaci D, et al
      . Dexamethasone versus Betamethasone for Preterm birth: a systematic review and network meta-analysis. Am J Obstet Gynecol MFM 2021;3:100312. doi:10.1016/j.ajogmf.2021.100312
      OpenUrl
    13. ↵
      1. Florez ID,
      2. Veroniki A-A,
      3. Al Khalifah R, et al
      . Comparative effectiveness and safety of interventions for acute diarrhea and gastroenteritis in children: a systematic review and network meta-analysis. PLoS One 2018;13:e0207701. doi:10.1371/journal.pone.0207701
    14. ↵
      1. Foote CJ,
      2. Guyatt GH,
      3. Vignesh KN, et al
      . Which surgical treatment for open Tibial shaft fractures results in the fewest Reoperations? A network meta-analysis. Clin Orthop Relat Res 2015;473:2179–92. doi:10.1007/s11999-015-4224-y
      OpenUrlCrossRefPubMed
    15. ↵
      1. Ge L,
      2. Sadeghirad B,
      3. Ball GDC, et al
      . Comparison of dietary Macronutrient patterns of 14 popular named dietary programmes for weight and cardiovascular risk factor reduction in adults: systematic review and network meta-analysis of randomised trials. BMJ 2020;369:m696. doi:10.1136/bmj.m696
    16. ↵
      1. Hanna C,
      2. Lawrie TA,
      3. Rogozińska E, et al
      . Treatment of newly diagnosed glioblastoma in the elderly: a network meta-analysis. Cochrane Database Syst Rev 2020;3:CD013261. doi:10.1002/14651858.CD013261.pub2
    17. ↵
      1. Isayama T,
      2. Iwami H,
      3. McDonald S, et al
      . Association of noninvasive ventilation strategies with mortality and Bronchopulmonary dysplasia among Preterm infants: a systematic review and meta-analysis. JAMA 2016;316:611–24. doi:10.1001/jama.2016.10708
      OpenUrlCrossRefPubMed
    18. ↵
      1. Jarde A,
      2. Lutsiv O,
      3. Park CK, et al
      . Effectiveness of progesterone, Cerclage and Pessary for preventing Preterm birth in Singleton pregnancies: a systematic review and network meta-analysis. BJOG 2017;124:1176–89. doi:10.1111/1471-0528.14624
      OpenUrlPubMed
    19. ↵
      1. Li G,
      2. Zeng J,
      3. Zhang J, et al
      . Comparative effects between direct oral anticoagulants for acute venous thromboembolism: indirect comparison from randomized controlled trials. Front Med (Lausanne) 2020;7:280. doi:10.3389/fmed.2020.00280
    20. ↵
      1. Liu J-W,
      2. Lin S-H,
      3. Wang L-C, et al
      . Comparison of antiviral agents for seasonal influenza outcomes in healthy adults and children: a systematic review and network meta-analysis. JAMA Netw Open 2021;4:e2119151. doi:10.1001/jamanetworkopen.2021.19151
    21. ↵
      1. Martins CC,
      2. Firmino RT,
      3. Riva JJ, et al
      . Desensitizing Toothpastes for Dentin hypersensitivity: a network meta-analysis. J Dent Res 2020;99:514–22. doi:10.1177/0022034520903036
      OpenUrl
    22. ↵
      1. McLeod SL,
      2. Brignardello-Petersen R,
      3. Worster A, et al
      . Comparative effectiveness of Antiarrhythmics for out-of-hospital cardiac arrest: a systematic review and network meta-analysis. Resuscitation 2017;121:90–7. doi:10.1016/j.resuscitation.2017.10.012
      OpenUrl
    23. ↵
      1. Naing C,
      2. Poovorawan Y,
      3. Tong KS
      . Comparative effectiveness of anti-viral drugs with dual activity for treating hepatitis B and HIV Co-infected patients: a network meta-analysis. BMC Infect Dis 2018;18:564. doi:10.1186/s12879-018-3506-x
    24. ↵
      1. Naing C,
      2. Whittaker MA,
      3. Htet NH, et al
      . Efficacy of Antimalarial drugs for treatment of uncomplicated Falciparum malaria in Asian region: a network meta-analysis. PLoS One 2019;14:e0225882. doi:10.1371/journal.pone.0225882
    25. ↵
      1. Niño-Serna LF,
      2. Acosta-Reyes J,
      3. Veroniki A-A, et al
      . Antiemetics in children with acute gastroenteritis: a meta-analysis. Pediatrics 2020;145:e20193260. doi:10.1542/peds.2019-3260
    26. ↵
      1. Park S-K,
      2. Lim T,
      3. Cho H, et al
      . Comparative effectiveness of pharmacological interventions to prevent postoperative delirium: a network meta-analysis. Sci Rep 2021;11:11922. doi:10.1038/s41598-021-91314-z
    27. ↵
      1. Schwingshackl L,
      2. Krause M,
      3. Schmucker C, et al
      . Impact of different types of olive oil on cardiovascular risk factors: a systematic review and network meta-analysis. Nutr Metab Cardiovasc Dis 2019;29:1030–9. doi:10.1016/j.numecd.2019.07.001
      OpenUrl
    28. ↵
      1. Schwingshackl L,
      2. Nitschke K,
      3. Zähringer J, et al
      . Impact of meal frequency on Anthropometric outcomes: a systematic review and network meta-analysis of randomized controlled trials. Adv Nutr 2020;11:1108–22. doi:10.1093/advances/nmaa056
      OpenUrl
    29. ↵
      1. Sekercioglu N,
      2. Angeliki Veroniki A,
      3. Thabane L, et al
      . Effects of different phosphate lowering strategies in patients with CKD on laboratory outcomes: a systematic review and NMA. PLoS One 2017;12:e0171028. doi:10.1371/journal.pone.0171028
    30. ↵
      1. Siemieniuk RA,
      2. Bartoszko JJ,
      3. Díaz Martinez JP, et al
      . Antibody and cellular therapies for treatment of COVID-19: a living systematic review and network meta-analysis. BMJ 2021;374:2231. doi:10.1136/bmj.n2231
      OpenUrl
    31. ↵
      1. Urquhart O,
      2. Tampi MP,
      3. Pilcher L, et al
      . Nonrestorative treatments for Caries: systematic review and network meta-analysis. J Dent Res 2019;98:14–26. doi:10.1177/0022034518800014
      OpenUrlCrossRef
    32. ↵
      1. Araújo EG,
      2. Oliveira DMSL de,
      3. Martins CC, et al
      . Efficacy of antioxidant supplementation to non-surgical Periodontal therapy on metabolic control in type 2 diabetes patients: a network meta-analysis. Antioxidants (Basel) 2022;11:621. doi:10.3390/antiox11040621
    33. ↵
      1. Facciorusso A,
      2. Ramai D,
      3. Gkolfakis P, et al
      . Comparative efficacy of different methods for difficult biliary Cannulation in ERCP: systematic review and network meta-analysis. Gastrointest Endosc 2022;95:60–71. doi:10.1016/j.gie.2021.09.010
      OpenUrl
    34. ↵
      1. Gao Y,
      2. Ge L,
      3. Liu M, et al
      . Comparative efficacy and acceptability of cognitive behavioral therapy delivery formats for insomnia in adults: a systematic review and network meta-analysis. Sleep Med Rev 2022;64:101648. doi:10.1016/j.smrv.2022.101648
      OpenUrlCrossRefPubMed
    35. ↵
      1. Gudivada KK,
      2. Kumar A,
      3. Sriram K, et al
      . Antioxidant Micronutrient supplements for adult critically ill patients: a Bayesian multiple treatment comparisons meta-analysis. Clin Nutr ESPEN 2022;47:78–88. doi:10.1016/j.clnesp.2021.12.015
      OpenUrl
    36. ↵
      1. Kazemi A,
      2. Ryul Shim S,
      3. Jamali N, et al
      . Comparison of nutritional supplements for Glycemic control in type 2 diabetes: a systematic review and network meta-analysis of randomized trials. Diabetes Res Clin Pract 2022;191:110037. doi:10.1016/j.diabres.2022.110037
      OpenUrl
    37. ↵
      1. Lewis SR,
      2. Macey R,
      3. Lewis J, et al
      . Surgical interventions for treating Extracapsular hip fractures in older adults: a network meta-analysis. Cochrane Database Syst Rev 2022;2:CD013405. doi:10.1002/14651858.CD013405.pub2
    38. ↵
      1. Pitre T,
      2. Kiflen M,
      3. Helmeczi W, et al
      . The comparative effectiveness of vasoactive treatments for Hepatorenal syndrome: a systematic review and network meta-analysis. Crit Care Med 2022;50:1419–29. doi:10.1097/CCM.0000000000005595
      OpenUrlCrossRef
    39. ↵
      1. Pitre T,
      2. Mah J,
      3. Helmeczi W, et al
      . Medical treatments for idiopathic pulmonary fibrosis: a systematic review and network meta-analysis. Thorax 2022;77:1243–50. doi:10.1136/thoraxjnl-2021-217976
      OpenUrlAbstract/FREE Full Text
    40. ↵
      1. Pitre T,
      2. Su J,
      3. Cui S, et al
      . Medications for the treatment of pulmonary arterial hypertension: a systematic review and network meta-analysis. Eur Respir Rev 2022;31:220036. doi:10.1183/16000617.0036-2022
    41. ↵
      1. Pitre T,
      2. Van Alstine R,
      3. Chick G, et al
      . Antiviral drug treatment for Nonsevere COVID-19: a systematic review and network meta-analysis. CMAJ 2022;194:E969–80. doi:10.1503/cmaj.220471
      OpenUrlAbstract/FREE Full Text
    42. ↵
      1. Tegegne TK,
      2. Rawstorn JC,
      3. Nourse RA, et al
      . Effects of exercise-based cardiac rehabilitation delivery modes on exercise capacity and health-related quality of life in heart failure: a systematic review and network meta-analysis. Open Heart 2022;9:e001949. doi:10.1136/openhrt-2021-001949
    43. ↵
      1. Zamani M,
      2. Alizadeh-Tabari S,
      3. Zamani V, et al
      . Worldwide and regional efficacy estimates of first-line Helicobacter Pylori treatments: a systematic review and network meta-analysis. J Clin Gastroenterol 2022;56:114–24. doi:10.1097/MCG.0000000000001641
      OpenUrl
    44. ↵
      1. Aoyama H,
      2. Uchida K,
      3. Aoyama K, et al
      . Assessment of therapeutic interventions and lung protective ventilation in patients with moderate to severe acute respiratory distress syndrome: a systematic review and network meta-analysis. JAMA Netw Open 2019;2:e198116. doi:10.1001/jamanetworkopen.2019.8116
    45. ↵
      1. Drucker AM,
      2. Ellis AG,
      3. Bohdanowicz M, et al
      . Systemic immunomodulatory treatments for patients with Atopic dermatitis: a systematic review and network meta-analysis. JAMA Dermatol 2020;156:659–67. doi:10.1001/jamadermatol.2020.0796
      OpenUrl
    46. ↵
      1. Hazlewood GS,
      2. Barnabe C,
      3. Tomlinson G, et al
      . Methotrexate monotherapy and methotrexate combination therapy with traditional and biologic disease modifying anti-rheumatic drugs for rheumatoid arthritis: a network meta-analysis. Cochrane Database Syst Rev 2016;2016:CD010227. doi:10.1002/14651858.CD010227.pub2
    47. ↵
      1. Jarde A,
      2. Lutsiv O,
      3. Beyene J, et al
      . Vaginal progesterone, oral progesterone, 17-OHPC, Cerclage, and Pessary for preventing Preterm birth in at-risk Singleton pregnancies: an updated systematic review and network meta-analysis. BJOG 2019;126:556–67. doi:10.1111/1471-0528.15566
      OpenUrlCrossRefPubMed
    48. ↵
      1. Sekercioglu N,
      2. Thabane L,
      3. Díaz Martínez JP, et al
      . Comparative effectiveness of phosphate binders in patients with chronic kidney disease: a systematic review and network meta-analysis. PLoS One 2016;11:e0156891. doi:10.1371/journal.pone.0156891
    49. ↵
      1. Zeng L,
      2. Qasim A,
      3. Neogi T, et al
      . Efficacy and safety of pharmacologic interventions in patients experiencing a gout flare: a systematic review and network meta-analysis. Arthritis Care & Research 2021;73:755–64. doi:10.1002/acr.24402 Available: https://acrjournals.onlinelibrary.wiley.com/toc/21514658/73/5
      OpenUrl
    50. ↵
      1. Lewis SR,
      2. Macey R,
      3. Stokes J, et al
      . Surgical interventions for treating Intracapsular hip fractures in older adults: a network meta-analysis. Cochrane Database Syst Rev 2022;2:CD013404. doi:10.1002/14651858.CD013404.pub2
    51. ↵
      1. Vosoughi K,
      2. Atieh J,
      3. Khanna L, et al
      . Association of glucagon-like peptide 1 analogs and agonists administered for obesity with weight loss and adverse events: a systematic review and network meta-analysis. EClinicalMedicine 2021;42:101213. doi:10.1016/j.eclinm.2021.101213
      OpenUrl
    52. ↵
      1. Katsanos K,
      2. Kitrou P,
      3. Spiliopoulos S, et al
      . Comparative effectiveness of plain balloon Angioplasty, bare metal Stents, drug-coated balloons, and drug-Eluting Stents for the treatment of Infrapopliteal artery disease. J Endovasc Ther 2016;23:851–63. doi:10.1177/1526602816671740
      OpenUrlCrossRefPubMed
    53. ↵
      1. Pagsberg AK,
      2. Tarp S,
      3. Glintborg D, et al
      . Acute antipsychotic treatment of children and adolescents with schizophrenia-spectrum disorders: a systematic review and network meta-analysis. J Am Acad Child Adolesc Psychiatry 2017;56:191–202. doi:10.1016/j.jaac.2016.12.013
      OpenUrl
    54. ↵
      1. Gallos ID,
      2. Papadopoulou A,
      3. Man R, et al
      . Uterotonic agents for preventing postpartum haemorrhage: a network meta-analysis. Cochrane Database Syst Rev 2018;12:CD011689. doi:10.1002/14651858.CD011689.pub3
    55. ↵
      1. Ghosh J,
      2. Papadopoulou A,
      3. Devall AJ, et al
      . Methods for managing Miscarriage: a network meta-analysis. Cochrane Database Syst Rev 2021;6:CD012602. doi:10.1002/14651858.CD012602.pub2
    56. ↵
      1. Kim DK,
      2. Kim JH,
      3. Lee JY, et al
      . Reappraisal of the treatment duration of antibiotic regimens for acute uncomplicated cystitis in adult women: a systematic review and network meta-analysis of 61 randomised clinical trials. Lancet Infect Dis 2020;20:1080–8. doi:10.1016/S1473-3099(20)30121-3
      OpenUrl
    57. ↵
      1. Solo K,
      2. Lavi S,
      3. Kabali C, et al
      . Antithrombotic treatment after coronary artery bypass graft surgery: systematic review and network meta-analysis. BMJ 2019;367:l5476. doi:10.1136/bmj.l5476
    58. ↵
      1. Srisurapanont K,
      2. Samakarn Y,
      3. Kamklong B, et al
      . Blue-wavelength light therapy for post-traumatic brain injury Sleepiness, sleep disturbance, depression, and fatigue: a systematic review and network meta-analysis. PLoS One 2021;16:e0246172. doi:10.1371/journal.pone.0246172
    59. ↵
      1. Srisurapanont M,
      2. Likhitsathian S,
      3. Suttajit S, et al
      . Efficacy and dropout rates of antipsychotic medications for methamphetamine psychosis: a systematic review and network meta-analysis. Drug Alcohol Depend 2021;219:108467. doi:10.1016/j.drugalcdep.2020.108467
      OpenUrlPubMed
    60. ↵
      1. Tarp S,
      2. Eric Furst D,
      3. Boers M, et al
      . Risk of serious adverse effects of biological and targeted drugs in patients with rheumatoid arthritis: a systematic review meta-analysis. Rheumatology (Oxford) 2017;56:417–25. doi:10.1093/rheumatology/kew442
      OpenUrlCrossRefPubMed
    61. ↵
      1. van den Houten MML,
      2. Hageman D,
      3. Gommans LNM, et al
      . The effect of supervised exercise, home based exercise and Endovascular Revascularisation on physical activity in patients with intermittent Claudication: a network meta-analysis. Eur J Vasc Endovasc Surg 2019;58:383–92. doi:10.1016/j.ejvs.2018.12.023
      OpenUrl
    62. ↵
      1. Health Quality Ontario
      . Left atrial appendage closure device with delivery system: a health technology assessment. Ont Health Technol Assess Ser 2017;17:1–106.
      OpenUrlPubMed
    63. ↵
      1. Burry LD,
      2. Cheng W,
      3. Williamson DR, et al
      . Pharmacological and non-pharmacological interventions to prevent delirium in critically ill patients: a systematic review and network meta-analysis. Intensive Care Med 2021;47:943–60. doi:10.1007/s00134-021-06490-3
      OpenUrlPubMed
    64. ↵
      1. Dreweck FDS,
      2. Burey A,
      3. de Oliveira Dreweck M, et al
      . Adhesive strategies in Cervical lesions: systematic review and a network meta-analysis of randomized controlled trials. Clin Oral Investig 2021;25:2495–510. doi:10.1007/s00784-021-03844-5
      OpenUrl
    65. ↵
      1. Katsanos K,
      2. Kitrou P,
      3. Spiliopoulos S, et al
      . Comparative effectiveness of plain balloon Angioplasty, bare metal Stents, drug-coated balloons, and drug-Eluting Stents for the treatment of Infrapopliteal artery disease: systematic review and Bayesian network meta-analysis of randomized controlled trials. J Endovasc Ther 2016;23:851–63. doi:10.1177/1526602816671740
      OpenUrlCrossRefPubMed
    66. ↵
      1. Paglia MDG,
      2. Silva MT,
      3. Lopes LC, et al
      . Use of Corticoids and non-Steroidal anti-Inflammatories in the treatment of rheumatoid arthritis: systematic review and network meta-analysis. PLoS One 2021;16:e0248866. doi:10.1371/journal.pone.0248866
    67. ↵
      1. Rodríguez-Perálvarez M,
      2. Guerrero-Misas M,
      3. Thorburn D, et al
      . Maintenance immunosuppression for adults undergoing liver transplantation: a network meta-analysis. Cochrane Database Syst Rev 2017;3:CD011639. doi:10.1002/14651858.CD011639.pub2
    68. ↵
      1. Zhang J,
      2. Sardana D,
      3. Li KY, et al
      . Topical fluoride to prevent root Caries: systematic review with network meta-analysis. J Dent Res 2020;99:506–13. doi:10.1177/0022034520906384
      OpenUrl
    69. ↵
      1. Paglia MDG,
      2. Silva MT,
      3. Lopes LC, et al
      . Use of Corticoids and non-Steroidal Antiinflammatories in the treatment of rheumatoid arthritis: systematic review and network meta-analysis. PLoS ONE 2021;16:e0248866. doi:10.1371/journal.pone.0248866
    70. ↵
      1. Grant S,
      2. Azhar G,
      3. Han E, et al
      . Clinical interventions for adults with comorbid alcohol use and depressive disorders: a systematic review and network meta-analysis. PLoS Med 2021;18:e1003822. doi:10.1371/journal.pmed.1003822
    71. ↵
      1. Ha V,
      2. Bonner AJ,
      3. Jadoo JK, et al
      . The effects of various diets on Glycemic outcomes during pregnancy: a systematic review and network meta-analysis. PLoS One 2017;12:e0182095. doi:10.1371/journal.pone.0182095
    72. ↵
      1. McBain C,
      2. Lawrie TA,
      3. Rogozińska E, et al
      . Treatment options for progression or recurrence of glioblastoma: a network meta-analysis. Cochrane Database Syst Rev 2021;5:CD013579. doi:10.1002/14651858.CD013579.pub2
    73. ↵
      1. Mocellin S,
      2. Goodwin A,
      3. Pasquali S
      . Risk-reducing medications for primary breast cancer: a network meta-analysis. Cochrane Database Syst Rev 2019;4:CD012191. doi:10.1002/14651858.CD012191.pub2
    74. ↵
      1. Moreno-Drada JA,
      2. Abreu LG,
      3. Lino PA, et al
      . Effectiveness of local Hemostatic to prevent bleeding in dental patients on anticoagulation: a systematic review and network meta-analysis. J Craniomaxillofac Surg 2021;49:570–83. doi:10.1016/j.jcms.2021.04.014
      OpenUrl
    75. ↵
      1. Pulikkotil SJ,
      2. Nagendrababu V,
      3. Veettil SK, et al
      . Effect of oral premedication on the anaesthetic efficacy of inferior alveolar nerve block in patients with irreversible Pulpitis - a systematic review and network meta-analysis of randomized controlled trials. Int Endod J 2018;51:989–1004. doi:10.1111/iej.12912
      OpenUrl
    76. ↵
      1. Sharma M,
      2. Singh S,
      3. Desai V, et al
      . Comparison of therapies for primary prevention of Esophageal Variceal bleeding: a systematic review and network meta-analysis. Hepatology 2019;69:1657–75. doi:10.1002/hep.30220
      OpenUrl
    77. ↵
      1. Singh JA,
      2. Hossain A,
      3. Mudano AS, et al
      . Biologics or tofacitinib for people with rheumatoid arthritis naive to methotrexate: a systematic review and network meta-analysis. Cochrane Database Syst Rev 2017;5:CD012657. doi:10.1002/14651858.CD012657
    78. ↵
      1. Singh JA,
      2. Hossain A,
      3. Tanjong Ghogomu E, et al
      . Biologics or tofacitinib for rheumatoid arthritis in incomplete responders to methotrexate or other traditional disease-modifying anti-rheumatic drugs: a systematic review and network meta-analysis. Cochrane Database Syst Rev 2016;2016:CD012183. doi:10.1002/14651858.CD012183
    79. ↵
      1. Singh JA,
      2. Hossain A,
      3. Tanjong Ghogomu E, et al
      . Biologic or tofacitinib monotherapy for rheumatoid arthritis in people with traditional disease-modifying anti-rheumatic drug (DMARD) failure: a Cochrane systematic review and network meta-analysis (NMA). Cochrane Database Syst Rev 2016;11:CD012437. doi:10.1002/14651858.CD012437
    80. ↵
      1. Singh S,
      2. Fumery M,
      3. Sandborn WJ, et al
      . Systematic review and network meta-analysis: first- and second-line biologic therapies for moderate-severe Crohn’s disease. Aliment Pharmacol Ther 2018;48:394–409. doi:10.1111/apt.14852 Available: https://onlinelibrary.wiley.com/toc/13652036/48/4
      OpenUrlPubMed
    81. ↵
      1. Tan-Lim CSC,
      2. Esteban-Ipac NAR,
      3. Recto MST, et al
      . Comparative effectiveness of Probiotic strains on the prevention of pediatric Atopic dermatitis: a systematic review and network meta-analysis. Pediatr Allergy Immunol 2021;32:1255–70. doi:10.1111/pai.13514
      OpenUrl
    82. ↵
      1. Torbahn G,
      2. Hofmann H,
      3. Rücker G, et al
      . Efficacy and safety of antibiotic therapy in early cutaneous Lyme Borreliosis: a network meta-analysis. JAMA Dermatol 2018;154:1292–303. doi:10.1001/jamadermatol.2018.3186
      OpenUrl
    83. ↵
      1. Xie D,
      2. Liao Y,
      3. Yue J, et al
      . Effects of five types of selenium supplementation for treatment of Kashin-Beck disease in children: a systematic review and network meta-analysis. BMJ Open 2018;8:e017883. doi:10.1136/bmjopen-2017-017883
    84. ↵
      1. Bae S,
      2. Alboog A,
      3. Esquivel KS, et al
      . Efficacy of perioperative pharmacological and regional pain interventions in adult spine surgery: a network meta-analysis and systematic review of randomised controlled trials. British Journal of Anaesthesia 2022;128:98–117. doi:10.1016/j.bja.2021.08.034
      OpenUrl
    85. ↵
      1. Sim R,
      2. Chong CW,
      3. Loganadan NK, et al
      . Comparative effectiveness of cardiovascular, renal and safety outcomes of second-line antidiabetic drugs use in people with type 2 diabetes: a systematic review and network meta-analysis of randomised controlled trials. Diabet Med 2022;39:e14780. doi:10.1111/dme.14780 Available: https://onlinelibrary.wiley.com/toc/14645491/39/3
    86. ↵
      1. Sandhu SS,
      2. Cheema MS,
      3. Khehra HS
      . Comparative effectiveness of pharmacologic and Nonpharmacologic interventions for orthodontic pain relief at peak pain intensity: a Bayesian network meta-analysis. Am J Orthod Dentofacial Orthop 2016;150:13–32. doi:10.1016/j.ajodo.2015.12.025
      OpenUrl
    87. ↵
      1. Schandelmaier S,
      2. Briel M,
      3. Varadhan R, et al
      . Development of the instrument to assess the credibility of effect modification analyses (ICEMAN) in randomized controlled trials and meta-analyses. CMAJ 2020;192:E901–6. doi:10.1503/cmaj.200077
      OpenUrlAbstract/FREE Full Text
    88. ↵
      1. Guyatt GH,
      2. Oxman AD,
      3. Kunz R, et al
      . GRADE guidelines: 8. rating the quality of evidence--Indirectness. J Clin Epidemiol 2011;64:1303–10. doi:10.1016/j.jclinepi.2011.04.014
      OpenUrlCrossRefPubMed
    89. ↵
      1. Brignardello-Petersen R,
      2. Mustafa RA,
      3. Siemieniuk RAC, et al
      . GRADE approach to rate the certainty from a network meta-analysis: addressing incoherence. J Clin Epidemiol 2019;108:77–85. doi:10.1016/j.jclinepi.2018.11.025
      OpenUrlCrossRefPubMed
    90. ↵

    Supplementary materials

    Footnotes

    • Contributors RBP, GG, BR and YW conceived the study idea and drafted the protocol. RCS, YF, RX, TP, GEB, LH, MMB, YG, MW and DG revised the protocol. YW, RX, TP, GEB, LH, MMB, YG, MW and DG conducted study selection and data extraction. YW analysed the data. YW, RBP and GG draft the manuscript. RX, TP, GEB, LH, MMB, YG, MW, DG, RAS, YF and BR reviewed, revised and approved this manuscript. YW is responsible for the overall content as guarantor.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.