You are here

  • Home
  • Archive
  • Volume 3, Issue 12
  • Poor adherence of randomised trials in surgery to CONSORT guidelines for non-pharmacological treatments (NPT): a cross-sectional study

Article Text

Article menu
XML
Medical publishing and peer review
Research
Poor adherence of randomised trials in surgery to CONSORT guidelines for non-pharmacological treatments (NPT): a cross-sectional study
  1. Myura Nagendran1,
  2. Daniel Harding2,
  3. Wendy Teo3,
  4. Christian Camm4,
  5. Mahiben Maruthappu5,
  6. Peter McCulloch6,
  7. Sally Hopewell7,8
  1. 1Green Templeton College, University of Oxford, Oxford, UK
  2. 2Magdalen College, University of Oxford, Oxford, UK
  3. 3Keble College, University of Oxford, Oxford, UK
  4. 4New College, University of Oxford, Oxford, UK
  5. 5Brigham and Women's Hospital, Center for Surgery and Public Health, Boston, USA
  6. 6Nuffield Department of Surgery, John Radcliffe Hospital, University of Oxford, Oxford, UK
  7. 7Centre for Statistics in Medicine, University of Oxford, Oxford, UK
  8. 8INSERM U738 Paris, France; Centre d’Épidémiologie Clinique, AP-HP (Assistance Publique des Hôpitaux de Paris), Hôpital Hôtel Dieu, Paris, France
  1. Correspondence to Dr Myura Nagendran; myura.nagendran{at}imperial.ac.uk

Abstract

Objective To systematically assess adherence of randomised trials in surgery to Consolidated Standards of Reporting Trials (CONSORT) guidelines for non-pharmacological treatments (NPT). Surgical trials are considered more difficult to design and execute than pharmacological trials. Furthermore, the original CONSORT statement does not address some aspects that are vital to the transparent reporting of surgical trials. The CONSORT-NPT extension was designed to address these issues but adherence in medical and surgical journals has not been assessed.

Design Cross-sectional study.

Sample We identified eight general medical and eight surgical journals, indexed in PubMed and published in 2011, with the highest impact factors in their respective categories.

Main outcomes Adherence to CONSORT statement and CONSORT-NPT extension items.

Results We identified 54 surgical trials (22 published in medical journals and 32 in surgical journals). There were eight items for which there was less than 30% overall compliance (seven were specific to the CONSORT-NPT extension). These seven items are related to: a full description of the care providers, centres and blinding status in the abstract (n=7/54, 13%), eligibility criteria for centres performing the interventions (n=13/54, 24%), how adherence of care providers with the protocol was assessed or enhanced (n=7/54, 13%), how clustering by care providers or centres was addressed as it relates to sample size (n=3/54, 6%), how care providers were allocated to each group (n=9/54, 17%), how clustering by care providers or centres was addressed as it relates to statistical methods (n=2/54, 4%), a description of care providers (case volume, qualification, expertise, etc) and centres (volume) in each group (n=0/54, 0%).

Conclusions Adherence of surgical trials to CONSORT-NPT extension items is much poorer than to the standard CONSORT statement. Adherence also appears to be superior in general medical journals compared with surgical journals. Raising awareness and conducting qualitative research to identify areas for specific intervention will be important going forward.

  • SURGERY

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/

https://doi.org/10.1136/bmjopen-2013-003898

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • This study is the first to assess surgical trials reported in general medical and surgical journals for adherence to the Consolidated Standards of Reporting Trials (CONSORT)-non-pharmacological treatments extension.

    • However, the final cross-sectional sample was small with only 54 trials. This precluded a detailed statistical analysis.

    • Furthermore, the cross-sectional nature of the study precluded us from determining whether there had been improvement over time.

    Introduction

    Randomised controlled trials (RCTs) are designed to determine the association between efficacy of a treatment and clinical outcome. In this regard, they are considered as the gold standard of healthcare evidence and the resulting conclusions can significantly affect clinical practice.1 It is therefore imperative that trials are well designed and correctly executed. However, it is equally important that trials are fully and transparently reported to allow proper critical appraisal by the scientific community.

    Key information is often missing from published trials2 ,3 and there may be a correlation between incomplete reporting and poor trial methodology.4–6 Such missing information can include items as crucial as sample size, details of randomisation, blinding and the choice of primary outcome. In response to this problem, the Consolidated Standards of Reporting Trials (CONSORT) statement was launched in 1996 and aimed to provide a checklist of essential items that authors should report when publishing their study.7 The CONSORT statement was updated in 2001 and recently in 2010 and is now endorsed by more than 600 leading medical journals.8 ,9 While the CONSORT statement has been credited with improving the reporting standards of RCTs,10 many recent studies have highlighted remaining deficiencies in medical11–14 and surgical literature.15–18

    Surgical trials are often considered more difficult to design and execute than pharmacological trials.19 Furthermore, the original CONSORT statement does not address some aspects that are vital to the transparent reporting of surgical trials such as difficulty in blinding patients and outcome assessors, variation in surgical technique and experience of operators. In 2008, an extension to the CONSORT statement was published providing specific recommendations for the reporting of RCTs of non-pharmacological treatment (CONSORT-NPT).20 Examples of added items include specifying the eligibility criteria for centres performing the intervention and how care providers are allocated to each trial group.

    The aim of this study was to analyse the quality of reporting of RCTs in surgery published in medical and surgical journals based on the reporting criteria included in the 2010 CONSORT statement and CONSORT-NPT extension.

    Methods

    Search strategy

    We identified eight general medical and eight surgical journals with the highest ISI impact factors from the ‘Medicine, General and Internal’ and ‘Surgery’ categories, respectively, of the 2011 Journal Citation Reports provided by Thomson Reuters.21 All 16 journals (see online supplementary appendix S1) are indexed on PubMed and a search was then conducted to identify reports of RCTs published in these 16 journals. The search (see online supplementary appendix S2) combined the ‘Cochrane Highly Sensitive Search Strategy’ for identifying randomised trials22 with the publication year 2011 and the journal name (conducted in March 2012). Also, the terms ‘surgery OR surgical OR surgeon’ were added when searching eight general medical journals to restrict results to RCTs in surgery. The search was conducted independently for each journal. All titles and abstracts retrieved from the search were assessed for eligibility by the authors (MN, MM, DH, WT and FC) such that each record was reviewed independently by at least two authors. Studies in which it was not clear whether the inclusion criteria had been met were reviewed in full text and discrepancies were resolved by consensus. All journals included in our sample are published in English.

    Inclusion and exclusion criteria

    We defined a randomised trial as a prospective study assessing healthcare interventions in human participants who were randomly allocated to study groups. Studies were considered eligible for inclusion if they were: (1) reports of a randomised controlled trial, (2) published in 2011 (either print or online e-publication during 2011) and (3) the primary aim of the study was considered as an interventional therapy. For the purposes of this study, an interventional therapy was defined as a therapy involving (1) some element of invasion or trauma to the body and 2) the requirement for operator skill to achieve a successful requirement and with the exception of an intervention being used purely to deliver a pharmacological treatment (ie, catheter delivered drug; see online supplementary appendix S2). We excluded reports where (1) one of the trial arms did not contain an interventional therapy as defined above, (2) a drug was the primary intervention, even in a surgical population (eg, chemotherapy for ovarian cancer) or (3) the RCT had been previously published and the current report was merely a follow-up or subgroup analysis using the same cohort of patients.

    Data extraction

    We created a modified version of the CONSORT checklist which contained all of the 2008 CONSORT-NPT checklist items and all of the standard 2010 CONSORT checklist items. The resulting checklist had a total of 42 items (see online supplementary appendix S3). Two authors (DH and WT) independently assessed each of the eligible reports against this checklist. Reports were also scored by the same authors for trial quality using the extended version of the Linde Internal Validity Scale (ELIVS) (see table 3, online supplementary appendix S2). The ELIVS scoring system used in this study was developed from initial work by Jadad et al23 and Linde et al.24 It measures the following quality domains: treatment allocation, randomisation method, allocation concealment, postrandomisation baseline comparison, blinding, handling and reporting of withdrawals and intention to treat analysis. Any discrepancies were resolved by consensus. Interobserver analysis was assessed by calculating the Cohen's κ score (score 0.74 based on disagreement of 268/2268 points). Extraction of data from studies was carried out in Microsoft Excel (2010, Microsoft Corporation, Redmond, Washington, USA) using a prepiloted form that was tested on two randomly selected studies from 2010.

    For each report, we also extracted the following data: the number of authors, the continent where the study was conducted, multicentre status, number of study participants and reporting of ethics review and conflict of interest. For each journal included, we obtained the ISI 2011 impact factor and whether or not the journal endorsed (eg, recommended or required) the CONSORT statement and CONSORT-NPT extension (information obtained in 2012).

    Author survey

    We also emailed the corresponding author for each included report, with five questions relating to the CONSORT-NPT extension in April 2013. The questions were: (1) Are you currently aware of the 2008 CONSORT-NPT extension? (2) Were you aware of the 2008 CONSORT-NPT extension at the time of submission? (3) Did the journal editorial staff mention the CONSORT-NPT extension to you during the editorial process (other than the instructions for authors on the journal website)? (4) Did the journal peer-reviewers mention the CONSORT-NPT extension to you during the review process? (5) Would your choice of journal for submission be affected by whether or not the journal mentions the CONSORT-NPT extension in their online instructions for authors? Each answer could be reported as ‘yes’, ‘no’ or ‘cannot remember/unsure’.

    Outcomes and statistical analysis

    Our primary outcome measure was adherence measured as the proportion of articles reporting each individual CONSORT and CONSORT-NPT checklist item. We also compared any differences in adherence between reports published in general medical journals with those published in surgical journals. All analyses were performed using STATA statistical software V.12.1 (College Station, Texas, USA).

    Results

    Our initial PubMed searches identified 831 possible reports, of which 771 were excluded as ineligible based on the information reported in the title and abstract. Sixty full-text articles were retrieved for further assessment, of which six were excluded because they were reports of previously published trials. This left 54 RCTs with a combined total of 16 338 patients from 11 journals that met the inclusion criteria (summarised in figure 1).

    Figure 1
    Figure 1

    Flow diagram of study selection.

    The baseline characteristics of the included trials are shown in table 1. The medical journals had a tendency towards a higher number of patients and a larger number of authors as well as a greater proportion of multicentre, higher quality (as measured by the ELIVS scale) trials. The requirement for CONSORT adherence was variable between the medical and surgical journals. Overall, only around half of the articles were published in a journal that required (26/54 studies; 48%) CONSORT adherence (table 1). The percentage of articles published in a journal that mentioned CONSORT in the instructions to peer reviewers (9/54 studies; 27%) was lower.

    View this table:
    Table 1

    Baseline characteristics of included studies

    Adherence of trials to the modified CONSORT checklist was variable ranging from 0% to 100% for each of the individual 42 checklist items (table 2). The highest scoring trials satisfied 36 of 42 items while the lowest scoring trial satisfied only 18 items (median 27, IQR 23–31). There were eight items for which there was less than 30% overall compliance (indicated with an asterisk in table 2). Of these eight items, seven were specific to the CONSORT-NPT extension. These seven items related to the following topics: a full description of the care providers, centres and blinding status in the abstract (item 1b; adherence 13%), eligibility criteria for centres performing the interventions (item 4b; adherence 24%), how adherence of care providers with the protocol was assessed or enhanced (item 5c; adherence 13%), how clustering by care providers or centres was addressed as it relates to sample size (item 7a; adherence 6%), how care providers were allocated to each group (item 8b; adherence 17%), how clustering by care providers or centres was addressed as it relates to statistical methods (item 12b; adherence 4%), a description of care providers (case volume, qualification, expertise, etc) and centres (volume) in each group (item 15b; adherence 0%). The non-CONSORT-NPT item with less than 30% adherence related to the presentation of absolute and relative effect sizes for binary outcomes (item 17b; adherence 28%).

    View this table:
    Table 2

    Adherence of studies to modified CONSORT-NPT checklist

    We did not compare the different adherence rates in a statistically formal way between the trials published in general medical and surgical journals as originally planned owing to the small sample size and the large number of hypotheses that could potentially be tested (all 42 checklist items). General medical journals tended to report better adherence to checklist items than surgical journals.

    We contacted the lead author for each of the 54 reports to ask about their awareness of CONSORT-NPT. Only 17 authors replied (31% response rate) and so we were not able to perform formal quantitative analysis on the survey results. On the basis of the replies we received, approximately one-third of respondents were aware of CONSORT-NPT at the time of submission, while two-thirds are aware of its existence now (table 3). Given the time lapse between manuscript submission and our short survey, about a quarter of respondents were unable to remember whether journal editors and peer-reviewers had mentioned CONSORT-NPT during the review process. Finally, one-third of respondents agreed that their choice of journal for submission would be affected by whether or not CONSORT-NPT was mentioned in the instructions for authors section of the journal (we did not ascertain the direction of this preference).

    View this table:
    Table 3

    Results from survey of corresponding authors

    Discussion

    Summary of main findings

    In this study, we included 54 reports of surgical RCTs published in 2011 from a cross-sectional sample of 11 medical and surgical journals. We assessed these reports for their adherence to a combined CONSORT and CONSORT-NPT checklist with two main findings. First, reporting adherence of surgical RCTs to the CONSORT-NPT extension was much poorer than adherence to the main CONSORT checklist. Second, general medical journals were broadly superior in their NPT reporting as compared with surgical journals. To our knowledge, this is the first study to demonstrate this difference between journal types for the CONSORT-NPT extension and one of only a few studies to document NPT adherence.

    Comparison with the literature

    The findings from our study are in agreement with the existing literature on CONSORT adherence. A recent systematic review of 53 studies found that reporting has remained suboptimal despite the CONSORT statement having been active in various iterations since 1996.25 However, the authors suggest that journal endorsement does appear to have had a positive impact on adherence. One review that included a comparison between surgical RCTs published in medical and surgical journals also found that adherence to CONSORT items was significantly superior in medical journals.16 Both of these articles assessed adherence only to the standard CONSORT statement. A recent study that assessed adherence specifically to the CONSORT-NPT extension checklist before (2004) and after (2010) the checklist was launched found little improvement in NPT-specific items (although these have had less time for absorption by the community than the standard CONSORT statement).17 These were reported in less than 50% of trials during 2010. The adherence rates in our study for similar NPT items were even lower.

    Limitations

    Our study has several limitations. First, we included only studies published in high-impact English language journals indexed in PubMed in 2011. In combination with our strict inclusion criteria on what constituted a surgical intervention, this led to a small sample of only 54 RCTs. We were consequently unable to perform a detailed statistical analysis on individual checklist items. The decision to limit the cross-sectional sample to 1 year was made on pragmatic grounds owing to the very lengthy process of scoring the RCTs against the checklist items. It was also for this reason that we restricted the number of journals we searched to the top eight impact factor journals within each specialty.

    A second limitation pertains to the cross-sectional nature of our study. We are unable to suggest whether any progress is being made in adherence to the NPT extension. As previously described, an interrupted before–after study found only moderate improvement between 2004 and 2010.17 Our author survey suggested that there had been an increase in the proportion of authors who were aware of the existence of the CONSORT-NPT extension between the time of submission and now. Whether this translates to improved adherence at the current time is unknown. A third limitation includes the fact that we did not assess study protocols for adherence to CONSORT criteria. Some authors may have included additional study details within the protocol.

    A final limitation concerns the author survey. We were restricted in our ability to analyse these data by the poor response rate and the significant time lag between submission of the RCTs (c. 2010) and distribution of the survey to the corresponding authors in 2013.

    Implications for authors and journals

    The CONSORT-NPT items with the poorest adherence were predominantly related to details on the implementation of the intervention, the providers (surgeons) and the centres. The CONSORT-NPT extension was specifically created to encourage reporting of these intervention-specific items given their importance in generalising a trial intervention to non-trial populations. As an example, two early symptomatic carotid surgery trials, NASCET and ACAS,26 ,27 had restrictive criteria for selecting which surgeons and centres were permitted to perform the intervention. Consequently, one large national cohort study that followed on from these trials did not see as large an improvement in patient outcomes.28 The study pointed out that less than 4% of all US hospitals providing carotid endarterectomy were included in NASCET and indeed that Medicare patients treated at trial hospitals had a lower risk of dying than at other hospitals.

    We might anecdotally expect CONSORT-NPT items to be more vigorously enforced by surgical journals. This is on the basis that surgeons (who would likely form a greater component of the journal's editorial board and peer reviewers) would be more familiar with the multiple elements of the intervention and the importance of these elements in their own practice. Naturally, therefore, they might be keener to see these reported more thoroughly in manuscripts reporting RCTs. Our results appear to suggest the opposite in that general medical journals displayed superior NPT adherence. It is difficult to ascertain whether this finding is confounded by the much larger impact factor of the general medical journals in our sample and the potential for a self-fulfilling prophecy (ie, better reported trials opt preferentially to try and publish in the high-impact medical journals rather than such RCTs being well reported as a prime result of enforcement by the medical journal). Overall, the wealth of potential confounders makes it difficult to conclude why medical journals displayed superior adherence.

    The literature on CONSORT adherence failures is extensive and is developing similarly for the CONSORT-NPT extension.25 Now that the problem has been well documented, the focus will likely shift towards identifying actionable areas for intervention. In the first instance, we suggest that qualitative interviews and focus groups with stakeholders at surgical trials departments will be important. Identifying the precise barriers to adherence will better inform the community on how best to improve reporting and where the greatest impact can be had. For example, does the problem lie with restrictive word counts, lack of time, lack of enforcement or simply just lack of awareness? How far would journals be prepared to go with enforcement? Would this be a viable option for smaller impact journals, perhaps fearful of driving authors away by enforcing reporting guidelines too rigidly?

    These are all important questions that could be further elucidated by qualitative research in this field. New guidance on surgical RCT methodology29 and calls for greater investment in surgical research30 should be combined with a greater awareness of the CONSORT-NPT extension. Reporting standards, like trial design, are not static but need to adapt to the changing research landscape. CONSORT, therefore, needs to respond to proposals for new reporting standards such as those proposed by the IDEAL Collaboration29 in future NPT extensions. Notably, the All Trials movement pushing for transparency of pharmaceutical trials has garnered much attention from the public and press over recent months.31 This momentum has added weight to the growing call for thorough reporting to be considered a core duty of clinical researchers rather than just a desirable trait.

    Conclusion

    The findings from this cross-sectional review of surgical trials suggest that adherence to CONSORT-NPT extension items is much poorer than to the standard CONSORT statement. Adherence also appears to be superior in general medical journals compared with surgical journals. A combination of more qualitative research to identify the areas for specific intervention and a continuing effort to raise awareness of the CONSORT-NPT extension among stakeholders will be important going forward.

    References

    1. ↵
      1. Abel U,
      2. Koch A
      . The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487–97.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Dickinson K,
      2. Bunn F,
      3. Wentz R,
      4. et al
      . Size and quality of randomised controlled trials in head injury: review of published studies. BMJ 2000;320:1308–11.
      OpenUrlAbstract/FREE Full Text
    3. ↵
      1. Thornley B,
      2. Adams C
      . Content and quality of 2000 controlled trials in schizophrenia over 50 years. BMJ 1998;317:1181–4.
      OpenUrlAbstract/FREE Full Text
    4. ↵
      1. Huwiler-Muntener K,
      2. Juni P,
      3. Junker C,
      4. et al
      . Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 2002;287:2801–4.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Juni P,
      2. Altman DG,
      3. Egger M
      . Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001;323:42–6.
      OpenUrlFREE Full Text
    6. ↵
      1. Montori VM,
      2. Wang YG,
      3. Alonso-Coello P,
      4. et al
      . Systematic evaluation of the quality of randomized controlled trials in diabetes. Diabetes Care 2006;29:1833–8.
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Begg C,
      2. Cho M,
      3. Eastwood S,
      4. et al
      . Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276:637–9.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Moher D,
      2. Hopewell S,
      3. Schulz KF,
      4. et al
      . CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869.
      OpenUrlFREE Full Text
    9. ↵
      1. Moher D,
      2. Schulz KF,
      3. Altman DG
      . The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001;357:1191–4.
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Turner L,
      2. Shamseer L,
      3. Altman DG,
      4. et al
      . Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev 2012;1:60.
      OpenUrlCrossRefPubMed
    11. ↵
      1. Devereaux PJ,
      2. Manns BJ,
      3. Ghali WA,
      4. et al
      . The reporting of methodological factors in randomized controlled trials and the association with a journal policy to promote adherence to the Consolidated Standards of Reporting Trials (CONSORT) checklist. Control Clin Trials 2002;23:380–8.
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Haidich AB,
      2. Birtsou C,
      3. Dardavessis T,
      4. et al
      . The quality of safety reporting in trials is still suboptimal: survey of major general medical journals. J Clin Epidemiol 2011;64:124–35.
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Mills EJ,
      2. Wu P,
      3. Gagnier J,
      4. et al
      . The quality of randomized trial reporting in leading medical journals since the revised CONSORT statement. Contemp Clin Trials 2005;26:480–7.
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Moher D,
      2. Jones A,
      3. Lepage L
      . Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA 2001;285:1992–5.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Agha R,
      2. Cooper D,
      3. Muir G
      . The reporting quality of randomised controlled trials in surgery: a systematic review. Int J Surg 2007;5:413–22.
      OpenUrlCrossRefPubMed
    16. ↵
      1. Balasubramanian SP,
      2. Wiener M,
      3. Alshameeri Z,
      4. et al
      . Standards of reporting of randomized controlled trials in general surgery: can we do better? Ann Surg 2006;244:663–7.
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Gray R,
      2. Sullivan M,
      3. Altman DG,
      4. et al
      . Adherence of trials of operative intervention to the CONSORT statement extension for non-pharmacological treatments: a comparative before and after study. Ann R Coll Surg Engl 2012;94:388–94.
      OpenUrlPubMed
    18. ↵
      1. Sinha S,
      2. Ashby E,
      3. Jayaram R,
      4. et al
      . Quality of reporting in randomized trials published in high-quality surgical journals. J Am Coll Surg 2009;209:565–71 e1.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. McCulloch P,
      2. Taylor I,
      3. Sasako M,
      4. et al
      . Randomised trials in surgery: problems and possible solutions. BMJ 2002; 324:1448–51.
      OpenUrlFREE Full Text
    20. ↵
      1. Boutron I,
      2. Moher D,
      3. Altman DG,
      4. et al
      . Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration. Ann Intern Med 2008;148:295–309.
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      Thomson-Reuters. ISI Web of Knowledge. 2012 [cited 18 Sep 2012]. http://wok.mimas.ac.uk/
    22. ↵
      1. Higgins JPT,
      2. Green S
      . Cochrane handbook for systematic reviews of interventions. The Cochrane Collaboration, 2011.
    23. ↵
      1. Jadad AR,
      2. Moore RA,
      3. Carroll D,
      4. et al
      . Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17:1–12.
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Linde K,
      2. Ramirez G,
      3. Mulrow CD,
      4. et al
      . St John's wort for depression—an overview and meta-analysis of randomised clinical trials. BMJ 1996;313:253–8.
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Turner L,
      2. Shamseer L,
      3. Altman DG,
      4. et al
      . Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 2012;11:MR000030.
      OpenUrlPubMed
    26. ↵
      North American Symptomatic Carotid Endarterectomy Trial Collaborators. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis. N Engl J Med 1991;325:445–53.
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      Executive Committee for the Asymptomatic Carotid Atherosclerosis Study. Endarterectomy for asymptomatic carotid artery stenosis. JAMA 1995;273:1421–8.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Wennberg DE,
      2. Lucas FL,
      3. Birkmeyer JD,
      4. et al
      . Variation in carotid endarterectomy mortality in the Medicare population: trial hospitals, volume, and patient characteristics. JAMA 1998; 279:1278–81.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Cook JA,
      2. McCulloch P,
      3. Blazeby JM,
      4. et al
      . IDEAL framework for surgical innovation 3: randomised controlled trials in the assessment stage and evaluations in the long term study stage. BMJ 2013;346:f2820.
      OpenUrlFREE Full Text
    30. ↵
      1. McCulloch P
      . How to improve surgical research. BMJ 2011;343:d4121.
      OpenUrlFREE Full Text
    31. ↵
      1. Brown T
      . It's time for AllTrials Registered and reported. Cochrane Database Syst Rev 2013;5:ED000057.
      OpenUrl

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors MN, CC and MM conceived the study. MN, CC, MM, PM and SH were involved in the design. MN, DH, WT, CC and MM independently identified studies for inclusion. DH and WT collected data from includable studies. MN analysed the data and wrote the first draft of the report. All authors contributed to subsequent drafts and approved the final version of the manuscript.

    • Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests SH, Centre for Statistics in Medicine, is a member of the CONSORT group. PM is the Chair of the IDEAL Collaboration.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement Electronic data collection sheets are available on request.