Protocol for the development of a reporting guideline for umbrella reviews on epidemiological associations using cross-sectional, case-control and cohort studies: the Preferred Reporting Items for Umbrella Reviews of Cross-sectional, Case-control and Cohort studies (PRIUR-CCC)
=====================================================================================================================================================================================================================================================================================

* Marco Solmi
* Kelly Cobey
* David Moher
* Sanam Ebrahimzadeh
* Elena Dragioti
* Jae Il Shin
* Joaquim Radua
* Samuele Cortese
* Beverley Shea
* Nicola Veronese
* Lisa Hartling
* Michelle Pollock
* Matthias Egger
* Stefania Papatheodorou
* John P Ioannidis
* Andre F Carvalho

## Abstract

**Introduction** Observational studies are fraught with several biases including reverse causation and residual confounding. Overview of reviews of observational studies (ie, umbrella reviews) synthesise systematic reviews with or without meta-analyses of cross-sectional, case-control and cohort studies, and may also aid in the grading of the credibility of reported associations. The number of published umbrella reviews has been increasing. Recently, a reporting guideline for overviews of reviews of healthcare interventions (Preferred Reporting Items for Overviews of Reviews (PRIOR)) was published, but the field lacks reporting guidelines for umbrella reviews of observational studies. Our aim is to develop a reporting guideline for umbrella reviews on cross-sectional, case-control and cohort studies assessing epidemiological associations.

**Methods and analysis** We will adhere to established guidance and prepare a PRIOR extension for systematic reviews of cross-sectional, case-control and cohort studies testing epidemiological associations between an exposure and an outcome, namely Preferred Reporting Items for Umbrella Reviews of Cross-sectional, Case-control and Cohort studies (PRIUR-CCC). Step 1 will be the project launch to identify stakeholders. Step 2 will be a literature review of available guidance to conduct umbrella reviews. Step 3 will be an online Delphi study sampling 100 participants among authors and editors of umbrella reviews. Step 4 will encompass the finalisation of PRIUR-CCC statement, including a checklist, a flow diagram, explanation and elaboration document. Deliverables will be (i) identifying stakeholders to involve according to relevant expertise and end-user groups, with an equity, diversity and inclusion lens; (ii) completing a narrative review of methodological guidance on how to conduct umbrella reviews, a narrative review of methodology and reporting in published umbrella reviews and preparing an initial PRIUR-CCC checklist for Delphi study round 1; (iii) preparing a PRIUR-CCC checklist with guidance after Delphi study; (iv) publishing and disseminating PRIUR-CCC statement.

**Ethics and dissemination** PRIUR-CCC has been approved by The Ottawa Health Science Network Research Ethics Board and has obtained consent (20220639-01H). Participants to step 3 will give informed consent. PRIUR-CCC steps will be published in a peer-reviewed journal and will guide reporting of umbrella reviews on epidemiological associations.

*   protocols & guidelines
*   statistics & research methods
*   epidemiologic studies

### STRENGTHS AND LIMITATIONS OF THIS STUDY

*   This protocol for reporting guidance of umbrella reviews of epidemiological associations is needed to address specific reporting challenges of observational studies.

*   This protocol follows the guidance for reporting checklist, which is standard in the field.

*   This study is limited from attrition rates, which is typical of Delphi studies.

## Introduction

There is evidence that the number of systematic reviews and meta-analyses in the literature has increased geometrically over the past two decades.1 2 Due to the increasing number of systematic reviews and meta-analyses on a given topic over the years, the field of knowledge synthesis has developed systematic reviews of systematic reviews, also called ‘reviews of reviews’, ‘overviews of (systematic) reviews’, ‘meta-reviews’ or ‘umbrella reviews’.3–8 Umbrella reviews can include interventional studies or observational studies.9 Overviews of reviews and umbrella reviews ideally aim to provide a comprehensive and systematic synthesis following the steps of a systematic review (ie, literature search, methodological quality appraisal, quantitative analysis where feasible and appropriate, etc) with systematic reviews as the unit of analysis.2 9 10 The field has seen a sharp increase in the number of published overviews of reviews and umbrella reviews over the past decade. For example, just limiting to umbrella reviews, approximately 56 were published in 2010, while 560 (10 times increase in yearly publications) were published in 2021 (PubMed (umbrella review)).

There is important variability between and within the approach of overviews of reviews and umbrella reviews, making results hardly comparable.11–13 This heterogeneity is not surprising, considering the large heterogeneity in the conception and implementation of both systematic reviews and meta-analyses.14 15 The methodology used to conduct systematic reviews and meta-analyses, as well as the quality of reporting, can affect conclusions and potentially lead to misleading interpretation of findings, misinforming policy makers, professional organisations and regulatory bodies, practitioners, patients, the public and other stakeholders. The quality and credibility of evidence synthesis efforts is also largely based on the quality of the credibility of their unit of inclusion (ie, individual studies). There is consensus that randomised controlled trials start from a higher credibility in the evidence-based medicine pyramid, and lower credibility is assigned to observational studies, which are more prone to bias. Interventional studies are not free from limitations, but in general experimental designs such as randomised controlled trials can protect from a number of biases, such as confounding by indication, or reverse causality. By contrast, observational evidence is prone to these and other biases, including excess of significance bias.16 Among observational studies, different study designs are adapted depending on the research question to be answered. For instance, studies measuring prognostic accuracy or prediction models are typically cohort studies that need internal development of the model, internal and external validation, calibration and accuracy measures.17 Cross-sectional studies can instead be used to investigate biomarkers or diagnostic accuracy of a given construct/test, or the prevalence of a disease. Other research questions, and typically epidemiological associations between two factors are generally explored with cohort, case-control and cross-sectional studies.18 19 More specifically, among studies investigating epidemiological associations, cross-sectional studies are typically used to measure the association between two factors, neglecting the direction of such association, while case-control and cohort studies are frequently used to measure associations between a construct of interest, and putative risk factors,1 14 20–23 or its outcomes,24 25 with the exposure occurring before the outcome. Research questions for which umbrella reviews of observational studies can be used are reported in table 1.

View this table:
[Table 1](http://bmjopen.bmj.com/content/14/6/e071136/T1)

Table 1 
Frequent research questions for which overviews of reviews of observational studies are used

Reporting guidelines, defined as ‘ checklist, flow diagram or explicit text to guide authors in reporting a specific type of research, developed using explicit methodology’,26 can be useful to improve the transparency, quality and reporting of individual studies, reviews or umbrella reviews. Interventional and observational evidence pose different methodological quality and reporting challenges, which are reflected from different reporting guidance for the two study designs. For observational evidence, several checklists of essential information to be reported in a paper are available for individual studies, systematic reviews and meta-analyses. For observational studies, the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network has disseminated Standards for Reporting Diagnostic accuracy studies,27 27 Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis28 and Strengthening the Reporting of Observational Studies in Epidemiology.29 For systematic reviews, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 (PRISMA 2020)30 is a broad guide inclusive of a range of primary study designs, and more specific statements are available, that is, the PRISMA for Diagnostic Test Accuracy,31 and PRISMA for reviews including harms outcomes,32 or the guidance on conducting systematic reviews and meta-analyses of observational studies on aetiology33 (table 2). The different checklists addressing different study designs and research questions well reflect their different methodological challenges, from original research to evidence synthesis.

View this table:
[Table 2](http://bmjopen.bmj.com/content/14/6/e071136/T2)

Table 2 
Key reporting guidelines across different research questions that can be addressed with observational studies

Regarding overview of reviews and umbrella reviews, virtually all reporting checklists proposed so far have focused on interventional evidence. The first proposed checklist34 was based on A MeaSurement Tool to Assess systematic Reviews quality assessment tool (AMSTAR)35 and Cochrane guidance.36 The same year, one further checklist for overviews of reviews37 merged evidence from PRISMA for abstracts,38 the Preferred reporting items for overviews of systematic reviews (PRIO) checklist39 and the Overview Quality Assessment Questionnaire.40 Then, a checklist for overviews of reviews41 was developed based on Cochrane recommendations,36 and the older versions of PRISMA42 (and not PRISMA 202030), and AMSTAR35 43 (and not AMSTAR 244). Later, checklist was developed for systematic reviews of reviews including harms, called PRIO,39 based on an older version of PRISMA,42 PRISMA harms32 and PRISMA for systematic review protocols.45 The same group also published a checklist for abstract of overviews of reviews of healthcare interventions.46 Recently, the Preferred Reporting Items for Systematic Reviews of Systematic Reviews/Meta-analyses (PRIOR) statement has been published to guide reporting of overviews of reviews of health interventions,3 adhering to EQUATOR guidance.26 In PRIOR protocol,3 authors acknowledged that several relevant sources exist that provide guidance on overviews of reviews or umbrella reviews, but they did not adhere to guidance endorsed by the EQUATOR Network.

Regarding umbrella reviews (ie, observational evidence investigating epidemiological associations), to the best of our knowledge no EQUATOR-adherent guidance has been developed, registered with or disseminated by EQUATOR group, nor any specific checklist has been previously proposed. Given that PRIOR focuses on interventional evidence, and that different reporting guidelines are needed for observational evidence (cross-sectional, case-control, cohort studies) on epidemiological associations versus interventional evidence, the aim of this project is to develop evidence-based and agreement-based guideline PRIOR-extension for reporting umbrella reviews (ie, cross-sectional, case-control, cohort studies testing epidemiological associations), adhering to established guidance26 and building on PRIOR statement.3 Specifically, this project will yield a Preferred Reporting Items for Umbrella Reviews of Cross-sectional, Case-control and Cohort studies (PRIUR-CCC), which will be published in a peer-reviewed journal and available via a dedicated website.

## Methods and analysis

### Transparency statement

We have also submitted this protocol to The Ottawa Health Science Network- Research Ethics Board and have obtained consent (20220639-01H). All participants to the Delphi survey will give written informed consent, which they will be able to withdraw at any time (yet anonymous responses cannot be withdrawn). This protocol is publicly available at medrxiv.org (MEDRXIV/2022/283572).

All study data and materials will be publicly available.

### Study design

This study will follow EQUATOR guidance for developing reporting checklists,26 and will be composed on four steps, namely project launch, literature review, Delphi survey and guideline statement preparation (figure 1).

![Figure 1](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/14/6/e071136/F1.medium.gif)

[Figure 1](http://bmjopen.bmj.com/content/14/6/e071136/F1)

Figure 1 
Development of PRIUR-CCC statement flow diagram. EQUATOR, Enhancing the QUAlity and Transparency Of health Research.

#### Project launch

Project launch will consist of reaching an agreement on roles and responsibilities of core team members (eg, identifying stakeholders that will participate in the Delphi study). A core group of researchers have prepared the protocol of the project. These authors will be the core team of the project, and have extensive record of umbrella reviews on cross-sectional, case-control and cohort studies investigating epidemiological associations. The project’s day-to-day steps and its finalisation will be responsibility of shared first authors, and last author. All other authors will contribute to the protocol, literature review, initial set of checklist items, Delphi survey and final PRIUR-CCC statement. The identification and involvement of stakeholders will follow the Practical Guidance for Involving Stakeholders in Health Research from the Multi Stakeholder Engagement Consortium.47 Reporting of involvement of patients and public will adhere to Guidance for Reporting Involvement of Patients and the Public 2 (GRIPP2).48

#### Literature narrative review

We will review the systematic reviews conducted in the context of PRIOR development regarding available guidance to conduct umbrella reviews. Then, since two systematic reviews of umbrella reviews have already been conducted by the members of this project,2 12 those two systematic reviews will be used to assess methodology and reporting of included umbrella reviews. Given that PRIUR-CCC will not focus on diagnostic test accuracy and prediction models, we will not cover those study designs as they are out of scope for PRIUR-CCC. After having reviewed PRIOR documentation and having reviewed the two previous systematic reviews of umbrella reviews, we will extract key methodological factors from identified umbrella reviews, and publish a review of methodology and reporting of previous umbrella reviews. The findings of this review will be used to identify PRIOR domains and items where changes are needed, and to produce an initial PRIOR checklist to start the Delphi study with.

#### Delphi survey

We will conduct a Delphi study. Delphi studies use social science survey techniques to structure communication between participants in order to drive consensus and make a collective decision.49 Typically, Delphi studies use several rounds of surveys in which participants, vote on issues. Between rounds, results of voting are aggregated and anonymised. They are then presented back to participants along with their own individual scores, and feedback on why others voted as they did.50 51 Among others, two strengths of this method include allowing for effective communication, limiting direct confrontation between individuals and giving participants the opportunity to consider the group’s thoughts and to compare and adjust their own score in the next round. *Participants*: we will aim to include 100 participants in Delphi study round 1, as done in PRIOR development.3 Participants to the Delphi study will meet any of the following criteria: (a) they have publication track of umbrella reviews in various fields, or (b) have publication track in Delphi surveys or (c) have publication track of reporting checklists of cross-sectional, case-control and cohort studies, or editors of peer-reviewed journals that published umbrella reviews on observational evidence or that have interest in umbrella reviews, or (d) funders of research or meta-research, or (e) practitioners, (f) policy makers, (g) evidence synthesis associations (eg, Cochrane, Campbell collaboration, Joanna Briggs Institute, others).3 Participants will be recruited via a two-step process. First, we have established a core group of participants based on criteria (a) or (b) or (c) described above (project launch), and past solid collaboration track record, that are authoring the present protocol. Second, additional stakeholders will be invited, according to (a) to (h) criteria above. *Delphi study methods*: this Delphi study will consist of three rounds. For the first two Delphi rounds, we will ask participants to complete an online survey which will be administered using Google Forms, structuring forms based on purpose-built platform for Delphi survey development and management.52 The third round will consist of a facilitated online consensus group meeting (using Zoom53).

##### Round 1

Age, gender, geographical area ([https://www.who.int/countries](https://www.who.int/countries)) and stakeholders group will be collected anonymously. Similar to what has been done for scoping reviews,54 we will build on existing PRIOR statement, which parallels PRISMA 202030 statement, for consistency and continuity with existing established reporting guidance for evidence syntheses. All participants will be asked what items of PRIOR3 will have to (i) remain unchanged, (ii) what will have to be changed and how, (iii) what will have to be removed (three answer options, one possible choice). In addition, participants will have the possibility to propose new items. As a starting point, the core team composed of authors of this protocol will provide a set of suggested items for participants to vote on. This starting set of items will be built based on experience in umbrella reviews. Participants will respond indicating their agreement/disagreement to have each item included in the PRISMA-CCC reporting guideline. Participants will be provided with a free-text box to fill in with additional comments to explain why they voted how they did or to propose wording amendments to the item, or new items. To decide to keep or remove a PRIOR item, or to add a new item, will require a minimum of 80% consensus among participants (based on findings from a systematic review of Delphi studies55). Items where an 80% consensus has not been reached, as well as changes to PRIOR items and new items will be voted on again in round 2. We anticipate the survey will take about 20 min to complete and will provide participants with a 3-week window to take part, with reminders sent after 1 and 2 weeks, respectively. We will pilot test the survey among the authors of the present protocol.

##### Round 2

All participants who completed round 1 of the Delphi will be re-invited to take part in round 2. Items which achieved consensus in round 1 will be shared with the participants. We will then ask participants to re-vote on any items that did not reach consensus or that were newly suggested. When re-voting on items that appeared in round 1, participants will also be provided with all comments provided by participants to justify their responses. Again, an 80% of consensus will be used to determine what items to include/exclude from the PRIUR-CCC guideline. If a new proposed item will be overlapping with existing items, Delphi moderator will add a comment to the new proposed item pointing to existing overlapping item.

##### Round 3

The core group and a purposeful selection of participants will be invited to round 3. We will aim to ensure representation from each of our geographically diverse institutions, each of the stakeholder groups and demographic variables including gender. Based on previous Delphi surveys and feasibility, we will invite no more than 30 participants in total to ensure feasibility to round 3.56 57 Round 3 will be moderated by core group members authoring the protocol, who will rotate every two items. We will present all participants with the results (ie, frequency of responses for each item, comments, changes and new items) of round 2 of the Delphi prior to the meeting and summarise these again briefly at the start of the consensus meeting. Participants will have the opportunity to discuss outstanding items one-by-one. They will then be asked to vote anonymously using real-time voting technology available via Zoom on each of these items. While diverse time zones will present challenges in other ways a virtual meeting may foster equity, diversity and inclusion of participants who might otherwise not have had funding or capacity to travel to an in-person event.

##### Analysis

For completers and drop-outs, demographics and responses will be presented using descriptive statistics using SPSS28. We will identify which items have and have not reached consensus for inclusion or exclusion based on our definition of 80% agreement and report this information for each Delphi round. The list of items identified for inclusion in PRIUR-CCC will be collated after round 3 and we will report the outcomes of participants ranking of these items in a table.

#### Timeline

The project has not started yet, and will be started in January 2024 and completed by December 2024.

#### Guidance statement

Co-first and last authors of this protocol will prepare the first draft of final PRIUR-CCC statement, that will then be approved after reiteration with other authors. This PRIUR-CCC statement will include the report of the whole project, the PRIUR-CCC checklist, the PRIUR-CCC flow diagram and an explanatory document that will inform on how to use PRIUR-CCC, with examples. PRIUR-CCC statement will be published in peer-reviewed journals, and on dedicated platforms. The Delphi process reporting will be informed by the Conducting and REporting of DElphi Studies checklist58 (table 3).

View this table:
[Table 3](http://bmjopen.bmj.com/content/14/6/e071136/T3)

Table 3 
CREDES checklist for survey studies

### Ethics and dissemination

PRIUR-CCC has been approved by The Ottawa Health Science Network-Research Ethics Board and have obtained consent (20220639-01H). Participants to step 3 will give informed consent. PRIUR-CCC steps will be published in a peer-reviewed journal and will guide reporting of umbrella reviews on epidemiological associations.

### Patient and public involvement

Reporting of involvement of patients and public will adhere to GRIPP2.48 All authors completing all three rounds of Delphi process and the core group will be invited to review and finally coauthor the publication.

## Discussion

This protocol for reporting guidance of umbrella reviews of epidemiological associations is needed to address specific reporting challenges of observational studies. PRIUR-CCC will provide a reporting framework to guide future umbrella reviews of observational studies assessing epidemiological associations, that can be used from researchers, reviewers, funders and editors to evaluate the transparency and quality of reporting of umbrella reviews, across different research questions. It will follow the guidance for reporting checklist, which are standard in the field. We propose that having reporting guidelines is of crucial relevance when included studies follow an observational design, with high baseline risk of confounding factors and numerous sources of bias potentially guiding misleading results.11 59 We aim to publish the report of this project including the final PRIUR-CCC guidelines. The main limitation of this study will be inherent in the Delphi study design, where only completers of all rounds fully contribute to the project.

## Ethics statements

### Patient consent for publication

Not applicable.

## Footnotes

*   X @dmoher, @CorteseSamuele, @eggersnsf

*   MS and KC contributed equally.

*   Contributors MS, KC, LH and AFC designed the protocol, drafted the first version of the protocol and finalised the final version. DM, SE, ED, JIIS, JR, SC, BS, NV, MP, ME, SP and JPI approved the initial study design, and revised multiple times the protocol and approved the final version.

*   Funding LH is supported by a Canada Research Chair in Knowledge Synthesis and Translation.LH is supported by a Canada Research Chair in Knowledge Synthesis and Translation.LH is supported by a Canada Research Chair in Knowledge Synthesis and Translation.

*   Competing interests MS received honoraria/has been a consultant for Angelini, Lundbeck, Otsuka.

*   Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the 'Methods' section for further details.

*   Provenance and peer review Not commissioned; externally peer reviewed.

[http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/).

## References

1.  Arango C, Dragioti E, Solmi M, et al. Risk and protective factors for mental disorders beyond genetics: an evidence-based Atlas. World Psychiatry 2021;20:417–36. [doi:10.1002/wps.20894](http://dx.doi.org/10.1002/wps.20894)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1002/wps.20894&link_type=DOI) 

2.  Janiaud P, Agarwal A, Tzoulaki I, et al. Validity of observational evidence on putative risk and protective factors: appraisal of 3744 meta-analyses on 57 topics. BMC Med 2021;19. [doi:10.1186/s12916-021-02020-6](http://dx.doi.org/10.1186/s12916-021-02020-6)
    
    

3.  Pollock M, Fernandes RM, Pieper D, et al. Preferred reporting items for overviews of reviews (PRIOR): a protocol for development of a reporting guideline for overviews of reviews of healthcare interventions. Syst Rev 2019;8:335. [doi:10.1186/s13643-019-1252-9](http://dx.doi.org/10.1186/s13643-019-1252-9)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1186/s13643-019-1252-9&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

4.  Bellou V, Belbasis L, Tzoulaki I, et al. Systematic evaluation of the associations between environmental risk factors and dementia: an umbrella review of systematic reviews and meta-analyses. Alzheimers Dement 2017;13:406–18. [doi:10.1016/j.jalz.2016.07.152](http://dx.doi.org/10.1016/j.jalz.2016.07.152)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jalz.2016.07.152&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

5.  Köhler CA, Evangelou E, Stubbs B, et al. Mapping risk factors for depression across the LifeSpan: an umbrella review of evidence from meta-analyses and Mendelian randomization studies. J Psychiatr Res 2018;103:189–207. [doi:10.1016/j.jpsychires.2018.05.020](http://dx.doi.org/10.1016/j.jpsychires.2018.05.020)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jpsychires.2018.05.020&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

6.  Solmi M, Köhler CA, Stubbs B, et al. Environmental risk factors and nonpharmacological and nonsurgical interventions for obesity: an umbrella review of meta ‐ analyses of cohort studies and randomized controlled trials. Eur J Clin Invest 2018;48:e12982. [doi:10.1111/eci.12982](http://dx.doi.org/10.1111/eci.12982)
    
    

7.  Correll CU, Rubio JM, Inczedy-Farkas G, et al. Efficacy of 42 pharmacologic cotreatment strategies added to antipsychotic monotherapy in schizophrenia: systematic overview and quality appraisal of the meta-analytic evidence. JAMA Psychiatry 2017;74:675–84. [doi:10.1001/jamapsychiatry.2017.0624](http://dx.doi.org/10.1001/jamapsychiatry.2017.0624)
    
    

8.  Papatheodorou S. Umbrella reviews: what they are and why we need them. Eur J Epidemiol 2019;34:543–6. [doi:10.1007/s10654-019-00505-6](http://dx.doi.org/10.1007/s10654-019-00505-6)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1007/s10654-019-00505-6&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

9.  Ioannidis JPA. Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ 2009;181:488–93. [doi:10.1503/cmaj.081086](http://dx.doi.org/10.1503/cmaj.081086)
    
    [FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo5OiIxODEvOC80ODgiO3M6NDoiYXRvbSI7czoyNjoiL2Jtam9wZW4vMTQvNi9lMDcxMTM2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10. Solmi M, Dragioti E, Croatto G, et al. Risk and protective factors for Cannabis, cocaine, and opioid use disorders: an umbrella review of meta-analyses of observational studies. Neurosci Biobehav Rev 2021;126:243–51. [doi:10.1016/j.neubiorev.2021.03.014](http://dx.doi.org/10.1016/j.neubiorev.2021.03.014)
    
    

11. Solmi M, Correll CU, Carvalho AF, et al. The role of meta-analyses and umbrella reviews in assessing the harms of psychotropic medications: beyond qualitative synthesis. Epidemiol Psychiatr Sci 2018;27:537–42. [doi:10.1017/S204579601800032X](http://dx.doi.org/10.1017/S204579601800032X)
    
    

12. Arango C, Dragioti E, Solmi M, et al. In press Evidence-based Atlas of risk and protective factors of mental disorders: meta-umbrella review. World Psychiatry 2021.
    
    

13. Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q 2016;94:485–514. [doi:10.1111/1468-0009.12210](http://dx.doi.org/10.1111/1468-0009.12210)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1111/1468-0009.12210&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=27620683&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

14. Solmi M, Köhler CA, Stubbs B, et al. Environmental risk factors and nonpharmacological and nonsurgical interventions for obesity: an umbrella review of meta-analyses of cohort studies and randomized controlled trials. Eur J Clin Invest 2018;48:e12982. [doi:10.1111/eci.12982](http://dx.doi.org/10.1111/eci.12982)
    
    

15. Hailes HP, Yu R, Danese A, et al. Long-term outcomes of childhood sexual abuse: an umbrella review. Lancet Psychiatry 2019;6:830–9. [doi:10.1016/S2215-0366(19)30286-X](http://dx.doi.org/10.1016/S2215-0366(19)30286-X)
    
    

16. Ioannidis JPA, Trikalinos TA. An exploratory test for an excess of significant findings. Clinical Trials 2007;4:245–53. [doi:10.1177/1740774507079441](http://dx.doi.org/10.1177/1740774507079441)
    
    

17. Fusar-Poli P, Rutigliano G, Stahl D, et al. Long-term validity of the at risk mental state (ARMS) for predicting psychotic and non-psychotic mental disorders. Eur Psychiatr 2017;42:49–54. [doi:10.1016/j.eurpsy.2016.11.010](http://dx.doi.org/10.1016/j.eurpsy.2016.11.010)
    
    

18. Guyatt GH, Oxman AD, Sultan S, et al. GRADE guidelines: 9. In: Rating up the quality of evidence, 64. 2011: 1311–6. [doi:10.1016/j.jclinepi.2011.06.004](http://dx.doi.org/10.1016/j.jclinepi.2011.06.004)
    
    

19. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924–6. [doi:10.1136/bmj.39489.470347.AD](http://dx.doi.org/10.1136/bmj.39489.470347.AD)
    
    [FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzYvNzY1MC85MjQiO3M6NDoiYXRvbSI7czoyNjoiL2Jtam9wZW4vMTQvNi9lMDcxMTM2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

20. Bortolato B, Köhler CA, Evangelou E, et al. Systematic assessment of environmental risk factors for bipolar disorder: an umbrella review of systematic reviews and meta-analyses. Bipolar Disord 2017;19:84–96. [doi:10.1111/bdi.12490](http://dx.doi.org/10.1111/bdi.12490)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

21. Solmi M, Radua J, Stubbs B, et al. Risk factors for eating disorders: an umbrella review of published meta-analyses. Braz J Psychiatry 2020;43:314–23. [doi:10.1590/1516-4446-2020-1099](http://dx.doi.org/10.1590/1516-4446-2020-1099)
    
    

22. Solmi M, Dragioti E, Croatto G, et al. Risk and protective factors for personality disorders: an umbrella review of published meta-analyses of case–control and cohort studies. Front Psychiatry 2021;12:679379. [doi:10.3389/fpsyt.2021.679379](http://dx.doi.org/10.3389/fpsyt.2021.679379)
    
    

23. Kim JY, Son MJ, Son CY, et al. Environmental risk factors and biomarkers for autism spectrum disorder: an umbrella review of the evidence. Lancet Psychiatry 2019;6:590–600. [doi:10.1016/S2215-0366(19)30181-6](http://dx.doi.org/10.1016/S2215-0366(19)30181-6)
    
    

24. Dragioti E, Solmi M, Favaro A, et al. Association of antidepressant use with adverse health outcomes: a systematic umbrella review. JAMA Psychiatry 2019;76:1241–55. [doi:10.1001/jamapsychiatry.2019.2859](http://dx.doi.org/10.1001/jamapsychiatry.2019.2859)
    
    

25. Li X, Meng X, Timofeeva M, et al. Serum uric acid levels and multiple health outcomes: umbrella review of evidence from observational studies, randomised controlled trials, and Mendelian randomisation studies. BMJ 2017;357:j2376. [doi:10.1136/bmj.j2376](http://dx.doi.org/10.1136/bmj.j2376)
    
    

26. Moher D, Schulz KF, Simera I, et al. Guidance for developers of health research reporting guidelines. PLoS Med 2010;7:e1000217. [doi:10.1371/journal.pmed.1000217](http://dx.doi.org/10.1371/journal.pmed.1000217)
    
    

27. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ 2015;351:h5527. [doi:10.1136/bmj.h5527](http://dx.doi.org/10.1136/bmj.h5527)
    
    

28. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Circulation 2015;131:211–9. [doi:10.1161/CIRCULATIONAHA.114.014508](http://dx.doi.org/10.1161/CIRCULATIONAHA.114.014508)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTQ6ImNpcmN1bGF0aW9uYWhhIjtzOjU6InJlc2lkIjtzOjk6IjEzMS8yLzIxMSI7czo0OiJhdG9tIjtzOjI2OiIvYm1qb3Blbi8xNC82L2UwNzExMzYuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

29. Elm E von, Altman DG, Egger M, et al. Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 2007;335:806–8. [doi:10.1136/bmj.39335.541782.AD](http://dx.doi.org/10.1136/bmj.39335.541782.AD)
    
    [FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzUvNzYyNC84MDYiO3M6NDoiYXRvbSI7czoyNjoiL2Jtam9wZW4vMTQvNi9lMDcxMTM2LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

30. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. [doi:10.1136/bmj.n71](http://dx.doi.org/10.1136/bmj.n71)
    
    

31. McInnes MDF, Moher D, Thombs BD, et al. Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: the PRISMA-DTA statement. JAMA 2018;319:388. [doi:10.1001/jama.2017.19163](http://dx.doi.org/10.1001/jama.2017.19163)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/jama.2017.19163&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=29362800&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

32. Zorzela L, Loke YK, Ioannidis JP, et al. PRISMA harms checklist: improving harms reporting in systematic reviews. BMJ 2016;352:i157. [doi:10.1136/bmj.i157](http://dx.doi.org/10.1136/bmj.i157)
    
    

33. Dekkers OM, Vandenbroucke JP, Cevallos M, et al. COSMOS-E: guidance on conducting systematic reviews and meta-analyses of observational studies of etiology. PLoS Med 2019;16:e1002742. [doi:10.1371/journal.pmed.1002742](http://dx.doi.org/10.1371/journal.pmed.1002742)
    
    

34. Singh J. Development of the metareview assessment of reporting quality (MARQ) checklist. Rev Fac Med 2012;60:285–92.
    
    

35. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10. [doi:10.1186/1471-2288-7-10](http://dx.doi.org/10.1186/1471-2288-7-10)
    
    

36. 1.  Higgins JSG
    
    Becker L, Oxman A. Chapter 22: Overviews of reviews. In: Higgins JSG, ed. Cochrane handbook for systematic reviews of interventions (version 5.1.0). London: The Cochrane Collaboration, 2011.
    
    

37. Li L, Tian J, Tian H, et al. Quality and transparency of overviews of systematic reviews. J Evid Based Med 2012;5:166–73. [doi:10.1111/j.1756-5391.2012.01185.x](http://dx.doi.org/10.1111/j.1756-5391.2012.01185.x)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1111/j.1756-5391.2012.01185.x&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

38. Beller EM, Glasziou PP, Altman DG, et al. PRISMA for abstracts: reporting systematic reviews in Journal and conference abstracts. PLoS Med 2013;10:e1001419. [doi:10.1371/journal.pmed.1001419](http://dx.doi.org/10.1371/journal.pmed.1001419)
    
    

39. Bougioukas KI, Liakos A, Tsapas A, et al. Preferred reporting items for overviews of systematic reviews including harms checklist: a pilot tool to be used for balanced reporting of benefits and harms. J Clin Epidemiol 2018;93:9–24. [doi:10.1016/j.jclinepi.2017.10.002](http://dx.doi.org/10.1016/j.jclinepi.2017.10.002)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2017.10.002&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=29037888&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

40. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol 1991;44:1271–8. [doi:10.1016/0895-4356(91)90160-b](http://dx.doi.org/10.1016/0895-4356(91)90160-b)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/0895-4356(91)90160-B&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=1834807&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1991GP82500017&link_type=ISI) 

41. 1.  Biondi-Zoccai G
    
    Onishi A, Furukawa T. Chapter 13: state-of-the-art reporting. In: Biondi-Zoccai G, ed. Umbrella reviews. Publishing, Cham: Springer International, 2016. [doi:10.1007/978-3-319-25655-9](http://dx.doi.org/10.1007/978-3-319-25655-9)
    
    

42. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009;6:e1000097. [doi:10.1371/journal.pmed.1000097](http://dx.doi.org/10.1371/journal.pmed.1000097)
    
    

43. Shea BJ, Bouter LM, Peterson J, et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One 2007;2:e1350. [doi:10.1371/journal.pone.0001350](http://dx.doi.org/10.1371/journal.pone.0001350)
    
    

44. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008. [doi:10.1136/bmj.j4008](http://dx.doi.org/10.1136/bmj.j4008)
    
    

45. Shamseer L, Moher D, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;350:g7647. [doi:10.1136/bmj.g7647](http://dx.doi.org/10.1136/bmj.g7647)
    
    

46. Bougioukas KI, Bouras E, Apostolidou-Kiouti F, et al. Reporting guidelines on how to write a complete and transparent abstract for overviews of systematic reviews of health care interventions. J Clin Epidemiol 2019;106:70–9. [doi:10.1016/j.jclinepi.2018.10.005](http://dx.doi.org/10.1016/j.jclinepi.2018.10.005)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2018.10.005&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

47. Concannon TW, Grant S, Welch V, et al. Practical guidance for involving stakeholders in health research. J Gen Intern Med 2019;34:458–63. [doi:10.1007/s11606-018-4738-6](http://dx.doi.org/10.1007/s11606-018-4738-6)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

48. Staniszewska S, Brett J, Simera I, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ 2017;358:j3453. [doi:10.1136/bmj.j3453](http://dx.doi.org/10.1136/bmj.j3453)
    
    

49. Linstone HA, Turoff M. Delphi: a brief look backward and forward. Technol Forecast Soc Change 2011;78:1712–9. [doi:10.1016/j.techfore.2010.09.011](http://dx.doi.org/10.1016/j.techfore.2010.09.011)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.techfore.2010.09.011&link_type=DOI) 

50. Dalkey N, Helmer O. An experimental application of the DELPHI method to the use of experts. Manag Sci 1963;9:458–67. [doi:10.1287/mnsc.9.3.458](http://dx.doi.org/10.1287/mnsc.9.3.458)
    
    

51. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm 2016;38:655–62. [doi:10.1007/s11096-016-0257-x](http://dx.doi.org/10.1007/s11096-016-0257-x)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1007/s11096-016-0257-x&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=26846316&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

52. Calibrum. Delphi surveys; n.d.
    
    

53. Zoom Video. Video conferencing, web conferecing, Webinars, screen sharing; n.d.
    
    

54. Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-SCR): checklist and explanation. Ann Intern Med 2018;169:467–73. [doi:10.7326/M18-0850](http://dx.doi.org/10.7326/M18-0850)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.7326/M18-0850&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=30178033&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

55. Diamond IR, Grant RC, Feldman BM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014;67:401–9. [doi:10.1016/j.jclinepi.2013.12.002](http://dx.doi.org/10.1016/j.jclinepi.2013.12.002)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2013.12.002&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=24581294&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

56. Cukier S, Lalu M, Bryson GL, et al. Defining predatory journals and responding to the threat they pose: a modified Delphi consensus process. BMJ Open 2020;10:e035561. [doi:10.1136/bmjopen-2019-035561](http://dx.doi.org/10.1136/bmjopen-2019-035561)
    
    

57. Fleuren M, Wiefferink K, Paulussen T. Determinants of innovation within health care organizations: literature review and Delphi study. Int J Qual Health Care 2004;16:107–23. [doi:10.1093/intqhc/mzh030](http://dx.doi.org/10.1093/intqhc/mzh030)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1093/intqhc/mzh030&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=15051705&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000220614500001&link_type=ISI) 

58. Jünger S, Payne SA, Brine J, et al. Guidance on conducting and reporting Delphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med 2017;31:684–706. [doi:10.1177/0269216317690685](http://dx.doi.org/10.1177/0269216317690685)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1177/0269216317690685&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

59. Munkholm K, Faurholt-Jepsen M, Ioannidis JPA, et al. Consideration of confounding was suboptimal in the reporting of observational studies in psychiatry: a meta-epidemiological study. J Clin Epidemiol 2020;119:75–84. [doi:10.1016/j.jclinepi.2019.12.002](http://dx.doi.org/10.1016/j.jclinepi.2019.12.002)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2019.12.002&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 

60. Benchimol EI, Smeeth L, Guttmann A, et al. The reporting of studies conducted using observational routinely-collected health data (RECORD) statement. PLoS Med 2015;12:e1001885. [doi:10.1371/journal.pmed.1001885](http://dx.doi.org/10.1371/journal.pmed.1001885)
    
    

61. Brooke BS, Schwartz TA, Pawlik TM. MOOSE reporting guidelines for meta-analyses of observational studies. JAMA Surg 2021;156:787–8. [doi:10.1001/jamasurg.2021.0522](http://dx.doi.org/10.1001/jamasurg.2021.0522)
    
    

62. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. J Am Med Assoc 2000;283:2008–12. [doi:10.1001/jama.283.15.2008](http://dx.doi.org/10.1001/jama.283.15.2008)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/jama.283.15.2008&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=10789670&link_type=MED&atom=%2Fbmjopen%2F14%2F6%2Fe071136.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000086436600037&link_type=ISI)