Intended for healthcare professionals

  1. Research
  2. Prediction models for...
  3. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal
CCBY Open access
Research

Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal

BMJ 2020; 369 doi: https://doi.org/10.1136/bmj.m1328 (Published 07 April 2020) Cite this as: BMJ 2020;369:m1328

Linked Editorial

Prediction models for diagnosis and prognosis in covid-19

Read our latest coverage of the coronavirus outbreak
  1. Laure Wynants, assistant professor1 2,
  2. Ben Van Calster, associate professor2 3,
  3. Gary S Collins, professor4 5,
  4. Richard D Riley, professor6,
  5. Georg Heinze, associate professor7,
  6. Ewoud Schuit, assistant professor8 9,
  7. Elena Albu, doctoral student2,
  8. Banafsheh Arshi, research fellow1,
  9. Vanesa Bellou, postdoctoral research fellow10,
  10. Marc M J Bonten, professor8 11,
  11. Darren L Dahly, principal statistician12 13,
  12. Johanna A Damen, assistant professor8 9,
  13. Thomas P A Debray, assistant professor8 14,
  14. Valentijn M T de Jong, assistant professor8 9,
  15. Maarten De Vos, associate professor2 15,
  16. Paula Dhiman, research fellow4 5,
  17. Joie Ensor, research fellow6,
  18. Shan Gao, doctoral student2,
  19. Maria C Haller, medical doctor7 16,
  20. Michael O Harhay, assistant professor17 18,
  21. Liesbet Henckaerts, assistant professor19 20,
  22. Pauline Heus, assistant professor8 9,
  23. Jeroen Hoogland, statistician8,
  24. Mohammed Hudda, senior research fellow21,
  25. Kevin Jenniskens, assistant professor8 9,
  26. Michael Kammer, research associate7 22,
  27. Nina Kreuzberger, research associate23,
  28. Anna Lohmann24,
  29. Brooke Levis, postdoctoral research fellow6,
  30. Kim Luijken, doctoral candidate24,
  31. Jie Ma, medical statistician5,
  32. Glen P Martin, senior lecturer25,
  33. David J McLernon, senior research fellow26,
  34. Constanza L Andaur Navarro, doctoral student8 9,
  35. Johannes B Reitsma, associate professor8 9,
  36. Jamie C Sergeant, senior lecturer27 28,
  37. Chunhu Shi, research associate29,
  38. Nicole Skoetz, professor22,
  39. Luc J M Smits, professor1,
  40. Kym I E Snell, senior lecturer6,
  41. Matthew Sperrin, senior lecturer30,
  42. René Spijker, information specialist8 9 31,
  43. Ewout W Steyerberg, professor3,
  44. Toshihiko Takada, associate professor8 32,
  45. Ioanna Tzoulaki, assistant professor10 33,
  46. Sander M J van Kuijk, research fellow34,
  47. Bas C T van Bussel, medical doctor1 35,
  48. Iwan C C van der Horst, professor35,
  49. Kelly Reeve36,
  50. Florien S van Royen, research fellow8,
  51. Jan Y Verbakel, assistant professor37 38,
  52. Christine Wallisch, research fellow7 39 40,
  53. Jack Wilkinson, research fellow24,
  54. Robert Wolff, medical doctor41,
  55. Lotty Hooft, professor8 9,
  56. Karel G M Moons, professor8 9,
  57. Maarten van Smeden, associate professor8
  1. 1Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands
  2. 2Department of Development and Regeneration, KU Leuven, Leuven, Belgium
  3. 3Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands
  4. 4Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Musculoskeletal Sciences, University of Oxford, Oxford, UK
  5. 5NIHR Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, UK
  6. 6Centre for Prognosis Research, School of Medicine, Keele University, Keele, UK
  7. 7Section for Clinical Biometrics, Centre for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Vienna, Austria
  8. 8Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
  9. 9Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands
  10. 10Department of Hygiene and Epidemiology, University of Ioannina Medical School, Ioannina, Greece
  11. 11Department of Medical Microbiology, University Medical Centre Utrecht, Utrecht, Netherlands
  12. 12HRB Clinical Research Facility, Cork, Ireland
  13. 13School of Public Health, University College Cork, Cork, Ireland
  14. 14Smart Data Analysis and Statistics BV, Utrecht, Netherlands
  15. 15Department of Electrical Engineering, ESAT Stadius, KU Leuven, Leuven, Belgium
  16. 16Ordensklinikum Linz, Hospital Elisabethinen, Department of Nephrology, Linz, Austria
  17. 17Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
  18. 18Palliative and Advanced Illness Research Center and Division of Pulmonary and Critical Care Medicine, Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
  19. 19Department of Microbiology, Immunology and Transplantation, KU Leuven-University of Leuven, Leuven, Belgium
  20. 20Department of General Internal Medicine, KU Leuven-University Hospitals Leuven, Leuven, Belgium
  21. 21Population Health Research Institute, St. George's University of London, Cranmer Terrace, London, UK
  22. 22Department of Nephrology, Medical University of Vienna, Vienna, Austria
  23. 23Evidence-Based Oncology, Department I of Internal Medicine and Centre for Integrated Oncology Aachen Bonn Cologne Dusseldorf, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany
  24. 24Department of Clinical Epidemiology, Leiden University Medical Centre, Leiden, Netherlands
  25. 25Division of Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK
  26. 26Institute of Applied Health Sciences, University of Aberdeen, Aberdeen, UK
  27. 27Centre for Biostatistics, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
  28. 28Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
  29. 29Division of Nursing, Midwifery and Social Work, School of Health Sciences, University of Manchester, Manchester, UK
  30. 30Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
  31. 31Amsterdam UMC, University of Amsterdam, Amsterdam Public Health, Medical Library, Netherlands
  32. 32Department of General Medicine, Shirakawa Satellite for Teaching And Research, Fukushima Medical University, Fukushima, Japan
  33. 33Department of Epidemiology and Biostatistics, Imperial College London School of Public Health, London, UK
  34. 34Department of Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Centre+, Maastricht, Netherlands
  35. 35Department of Intensive Care Medicine, Maastricht University Medical Centre+, Maastricht University, Maastricht, Netherlands
  36. 36Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Zurich, CH
  37. 37EPI-Centre, Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium
  38. 38NIHR Community Healthcare Medtech and IVD cooperative, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
  39. 39Charité Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-Universität zu Berlin, Berlin, Germany
  40. 40Berlin Institute of Health, Berlin, Germany
  41. 41Kleijnen Systematic Reviews, York, UK
  1. Correspondence to: L Wynants laure.wynants{at}maastrichtuniversity.nl
  • Accepted 31 March 2020
  • Final version accepted 17 July 2022

Abstract

Objective To review and appraise the validity and usefulness of published and preprint reports of prediction models for prognosis of patients with covid-19, and for detecting people in the general population at increased risk of covid-19 infection or being admitted to hospital or dying with the disease.

Design Living systematic review and critical appraisal by the covid-PRECISE (Precise Risk Estimation to optimise covid-19 Care for Infected or Suspected patients in diverse sEttings) group.

Data sources PubMed and Embase through Ovid, up to 17 February 2021, supplemented with arXiv, medRxiv, and bioRxiv up to 5 May 2020.

Study selection Studies that developed or validated a multivariable covid-19 related prediction model.

Data extraction At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool).

Results 126 978 titles were screened, and 412 studies describing 731 new prediction models or validations were included. Of these 731, 125 were diagnostic models (including 75 based on medical imaging) and the remaining 606 were prognostic models for either identifying those at risk of covid-19 in the general population (13 models) or predicting diverse outcomes in those individuals with confirmed covid-19 (593 models). Owing to the widespread availability of diagnostic testing capacity after the summer of 2020, this living review has now focused on the prognostic models. Of these, 29 had low risk of bias, 32 had unclear risk of bias, and 545 had high risk of bias. The most common causes for high risk of bias were inadequate sample sizes (n=408, 67%) and inappropriate or incomplete evaluation of model performance (n=338, 56%). 381 models were newly developed, and 225 were external validations of existing models. The reported C indexes varied between 0.77 and 0.93 in development studies with low risk of bias, and between 0.56 and 0.78 in external validations with low risk of bias. The Qcovid models, the PRIEST score, Carr’s model, the ISARIC4C Deterioration model, and the Xie model showed adequate predictive performance in studies at low risk of bias. Details on all reviewed models are publicly available at https://www.covprecise.org/.

Conclusion Prediction models for covid-19 entered the academic literature to support medical decision making at unprecedented speed and in large numbers. Most published prediction model studies were poorly reported and at high risk of bias such that their reported predictive performances are probably optimistic. Models with low risk of bias should be validated before clinical implementation, preferably through collaborative efforts to also allow an investigation of the heterogeneity in their performance across various populations and settings. Methodological guidance, as provided in this paper, should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, prediction modellers should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline.

Systematic review registration Protocol https://osf.io/ehc47/, registration https://osf.io/wy245.

Readers’ note This article is the final version of a living systematic review that has been updated over the past two years to reflect emerging evidence. This version is update 4 of the original article published on 7 April 2020 (BMJ 2020;369:m1328). Previous updates can be found as data supplements (https://www.bmj.com/content/369/bmj.m1328/related#datasupp). When citing this paper please consider adding the update number and date of access for clarity.

Footnotes

  • Contributors: LW conceived the study. LW and MvS designed the study. LW, MvS, and BVC screened titles and abstracts for inclusion. LW, BVC, GSC, TPAD, MCH, GH, KGMM, RDR, ES, LJMS, EWS, KIES, CW, AL, JM, TT, JAD, KL, JBR, LH, CS, MS, MCH, NS, NK, SMJvK, JCS, PD, CLAN, RW, GPM, IT, JYV, DLD, JW, FSvR, PH, VMTdJ, BCTvB, ICCvdH, DJM, MK, BL, EA, SG, BA, JH, KJ, SG, KR, JE, MH, VB, and MvS extracted and analysed data. MDV helped interpret the findings on deep learning studies and MMJB, LH, and MCH assisted in the interpretation from a clinical viewpoint. RS and FSvR offered technical and administrative support. LW wrote the first draft, which all authors revised for critical content. All authors approved the final manuscript. LW and MvS are the guarantors. The guarantors had full access to all the data in the study, take responsibility for the integrity of the data and the accuracy of the data analysis, and had final responsibility for the decision to submit for publication. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: LW, BVC, LH, and MDV acknowledge specific funding for this work from Internal Funds KU Leuven, KOOR, and the covid-19 Fund. LW is a postdoctoral fellow of Research Foundation-Flanders (FWO) and receives support from ZonMw (grant 10430012010001). BVC received support from FWO (grant G0B4716N) and Internal Funds KU Leuven (grant C24/15/037). TPAD acknowledges financial support from the Netherlands Organisation for Health Research and Development (grant 91617050). VMTdJ was supported by the European Union Horizon 2020 Research and Innovation Programme under ReCoDID grant agreement 825746. KGMM and JAD acknowledge financial support from Cochrane Collaboration (SMF 2018). KIES is funded by the National Institute for Health Research (NIHR) School for Primary Care Research. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care. GSC was supported by the NIHR Biomedical Research Centre, Oxford, and Cancer Research UK (programme grant C49297/A27294). JM was supported by the Cancer Research UK (programme grant C49297/A27294). PD was supported by the NIHR Biomedical Research Centre, Oxford. MOH is supported by the National Heart, Lung, and Blood Institute of the United States National Institutes of Health (grant R00 HL141678). ICCvDH and BCTvB received funding from Euregio Meuse-Rhine (grant Covid Data Platform (coDaP) interreg EMR-187). BL was supported by a Fonds de recherche du Québec-Santé postdoctoral training fellowship. JYV acknowledges the National Institute for Health and Care Research (NIHR) Community Healthcare MedTech and In Vitro Diagnostics Co-operative at Oxford Health NHS Foundation Trust. The funders had no role in study design, data collection, data analysis, data interpretation, or reporting.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: support from Internal Funds KU Leuven, KOOR, and the covid-19 Fund for the submitted work; no competing interests with regards to the submitted work; LW discloses support from Research Foundation-Flanders; RDR reports personal fees as a statistics editor for The BMJ (since 2009), consultancy fees for Roche for giving meta-analysis teaching and advice in October 2018, and personal fees for delivering in-house training courses at Barts and the London School of Medicine and Dentistry, and the Universities of Aberdeen, Exeter, and Leeds, all outside the submitted work; MS coauthored the editorial on the original article.

  • The lead authors affirm that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.

  • Dissemination to participants and related patient and public communities: The authors and patient partners will distribute this information through their institutions and on social media to provide an opportunity for public dialogue and as an example of how what we learn as the result of a new disease changes and improves over time. The study protocol is available online at https://osf.io/ehc47/.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

Data availability statement

The study protocol is available online at https://osf.io/ehc47/. Detailed extracted data on all included studies are available on https://www.covprecise.org/.

http://creativecommons.org/licenses/by/4.0/

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

View Full Text
Back to top