Intended for healthcare professionals

  1. Research
  2. Clinical prediction...
  3. Clinical prediction models for mortality in patients with covid-19: external validation and individual participant data meta-analysis
CCBY Open access
Research

Clinical prediction models for mortality in patients with covid-19: external validation and individual participant data meta-analysis

BMJ 2022; 378 doi: https://doi.org/10.1136/bmj-2021-069881 (Published 12 July 2022) Cite this as: BMJ 2022;378:e069881
  1. Valentijn M T de Jong, assistant professor1 2 3,
  2. Rebecca Z Rousset, masters student1,
  3. Neftalí Eduardo Antonio-Villa, research assistant45,
  4. Arnoldus G Buenen, emergency physician6 7,
  5. Ben Van Calster, associate professor8 9 10,
  6. Omar Yaxmehen Bello-Chavolla, assistant professor4,
  7. Nigel J Brunskill, professor11 12,
  8. Vasa Curcin, reader13,
  9. Johanna A A Damen, assistant professor1 2,
  10. Carlos A Fermín-Martínez, doctoral candidate4 5,
  11. Luisa Fernández-Chirino, professor and research assistant4 14,
  12. Davide Ferrari, doctoral student13 15,
  13. Robert C Free, research fellow16 17,
  14. Rishi K Gupta, senior mentor18,
  15. Pranabashis Haldar, clinical senior lecturer16 17 19,
  16. Pontus Hedberg, doctoral candidate20 21,
  17. Steven Kwasi Korang, medical doctor22,
  18. Steef Kurstjens, medical resident23,
  19. Ron Kusters, professor23 24,
  20. Rupert W Major, honorary associate professor11 25,
  21. Lauren Maxwell, senior researcher26,
  22. Rajeshwari Nair, research faculty27 28,
  23. Pontus Naucler, associate professor20 21,
  24. Tri-Long Nguyen, assistant professor1 29 30,
  25. Mahdad Noursadeghi, infectious diseases consultant31,
  26. Rossana Rosa, infectious diseases consultant32,
  27. Felipe Soares, doctoral candidate33,
  28. Toshihiko Takada, associate professor1 34,
  29. Florien S van Royen, doctoral candidate1,
  30. Maarten van Smeden, associate professor1,
  31. Laure Wynants, assistant professor7 35,
  32. Martin Modrák, postdoctoral researcher36,
  33. the CovidRetro collaboration,
  34. Folkert W Asselbergs, professor37 38 39,
  35. Marijke Linschoten, medical doctor and doctoral candidate37,
  36. CAPACITY-COVID consortium,
  37. Karel G M Moons, professor1 2,
  38. Thomas P A Debray, assistant professor1 2
  1. 1Julius Center for Health Sciences and Primary Care, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
  2. 2Cochrane Netherlands, University Medical Centre Utrecht, Utrecht University, Netherlands
  3. 3Data Analytics and Methods Task Force, European Medicines Agency, Amsterdam, Netherlands
  4. 4Dirección de Investigación, Instituto Nacional de Geriatría, Mexico City, Mexico
  5. 5MD/PhD (PECEM) Program, Faculty of Medicine, National Autonomous University of Mexico, Mexico City, Mexico
  6. 6Maxima MC, Veldhoven, the Netherlands
  7. 7Bernhoven, Uden, Netherlands
  8. 8Department of Development and Regeneration, KU Leuven, Leuven, Belgium
  9. 9Department of Biomedical Data Sciences, Leiden University Medical Centre, Leiden, Netherlands
  10. 10EPI-centre, KU Leuven, Leuven, Belgium
  11. 11Department of Cardiovascular Sciences, College of Life Sciences, University of Leicester, Leicester, UK
  12. 12John Walls Renal Unit, University Hospitals of Leicester NHS Trust, Leicester, UK
  13. 13School of Population Health and Environmental Sciences, King’s College London, London, UK
  14. 14Faculty of Chemistry, Universidad Nacional Autónoma de México, México City, Mexico
  15. 15Centre for Clinical Infection and Diagnostics Research, School of Immunology and Microbial Sciences, King’s College London, London, UK
  16. 16Department of Respiratory Sciences, College of Life Sciences, University of Leicester, Leicester, UK
  17. 17NIHR Leicester Biomedical Research Centre, University of Leicester, Leicester, UK
  18. 18Institute for Global Health, University College London, London, UK
  19. 19Department of Respiratory Medicine, University Hospitals of Leicester NHS Trust, Leicester, UK
  20. 20Department of Infectious Diseases, Karolinska University Hospital, Stockholm, Sweden
  21. 21Division of Infectious Diseases, Department of Medicine Solna, Karolinska Institute, Stockholm, Sweden
  22. 22Copenhagen Trial Unit, Centre for Clinical Intervention Research, Department 7812, Rigshospitalet, Copenhagen University Hospital, Denmark
  23. 23Laboratory of Clinical Chemistry and Haematology, Jeroen Bosch Hospital, Den Bosch, Netherlands
  24. 24Department of Health Technology and Services Research, Technical Medical Centre, University of Twente, Enschede, Netherlands
  25. 25Department of Cardiovascular Sciences, College of Life Sciences, University of Leicester, Leicester, UK
  26. 26Heidelberger Institut für Global Health, Universitätsklinikum Heidelberg, Germany
  27. 27University of Iowa Carver College of Medicine, Iowa City, IA, USA
  28. 28Centre for Access and Delivery Research Evaluation Iowa City Veterans Affairs Health Care System, Iowa City, IA, USA
  29. 29Section of Epidemiology, Department of Public Health, University of Copenhagen, Copenhagen, Denmark
  30. 30Department of Pharmacy, University Hospital Centre of Nîmes, Nîmes, France
  31. 31Division of Infection and Immunity, University College London, London, UK
  32. 32Infectious Diseases Service, UnityPoint Health-Des Moines, Des Moines, IA, USA
  33. 33Industrial Engineering Department, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
  34. 34Department of General Medicine, Shirakawa Satellite for Teaching And Research (STAR), Fukushima Medical University, Fukushima, Japan
  35. 35Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands
  36. 36Institute of Microbiology of the Czech Academy of Sciences, Prague, Czech Republic
  37. 37Department of Cardiology, Division of Heart and Lungs, University Medical Centre Utrecht, Utrecht University, Utrecht, Netherlands
  38. 38Health Data Research UK and Institute of Health Informatics, University College London, London, UK
  39. 39Institute of Cardiovascular Science, Faculty of Population Health Sciences, University College London, London, UK
  1. Correspondence to: V M T de Jong V.M.T.deJong-2{at}umcutrecht.nl
  • Accepted 25 May 2022

Abstract

Objective To externally validate various prognostic models and scoring rules for predicting short term mortality in patients admitted to hospital for covid-19.

Design Two stage individual participant data meta-analysis.

Setting Secondary and tertiary care.

Participants 46 914 patients across 18 countries, admitted to a hospital with polymerase chain reaction confirmed covid-19 from November 2019 to April 2021.

Data sources Multiple (clustered) cohorts in Brazil, Belgium, China, Czech Republic, Egypt, France, Iran, Israel, Italy, Mexico, Netherlands, Portugal, Russia, Saudi Arabia, Spain, Sweden, United Kingdom, and United States previously identified by a living systematic review of covid-19 prediction models published in The BMJ, and through PROSPERO, reference checking, and expert knowledge.

Model selection and eligibility criteria Prognostic models identified by the living systematic review and through contacting experts. A priori models were excluded that had a high risk of bias in the participant domain of PROBAST (prediction model study risk of bias assessment tool) or for which the applicability was deemed poor.

Methods Eight prognostic models with diverse predictors were identified and validated. A two stage individual participant data meta-analysis was performed of the estimated model concordance (C) statistic, calibration slope, calibration-in-the-large, and observed to expected ratio (O:E) across the included clusters.

Main outcome measures 30 day mortality or in-hospital mortality.

Results Datasets included 27 clusters from 18 different countries and contained data on 46 914patients. The pooled estimates ranged from 0.67 to 0.80 (C statistic), 0.22 to 1.22 (calibration slope), and 0.18 to 2.59 (O:E ratio) and were prone to substantial between study heterogeneity. The 4C Mortality Score by Knight et al (pooled C statistic 0.80, 95% confidence interval 0.75 to 0.84, 95% prediction interval 0.72 to 0.86) and clinical model by Wang et al (0.77, 0.73 to 0.80, 0.63 to 0.87) had the highest discriminative ability. On average, 29% fewer deaths were observed than predicted by the 4C Mortality Score (pooled O:E 0.71, 95% confidence interval 0.45 to 1.11, 95% prediction interval 0.21 to 2.39), 35% fewer than predicted by the Wang clinical model (0.65, 0.52 to 0.82, 0.23 to 1.89), and 4% fewer than predicted by Xie et al’s model (0.96, 0.59 to 1.55, 0.21 to 4.28).

Conclusion The prognostic value of the included models varied greatly between the data sources. Although the Knight 4C Mortality Score and Wang clinical model appeared most promising, recalibration (intercept and slope updates) is needed before implementation in routine care.

Footnotes

  • Contributors: FvR, JD, MvS, TT, KM, VdJ, TD, BVC, and LW were responsible for the systematic review and design of the study. VdJ and TD were responsible for the statistical analysis plan and R code. FWA, OB-C, VC, RZR, FS, YY, TT, PN, PH, SK, RK, ML, RKG, MN, LFCM, AB, CAPACITY-COVID consortium (see supplementary material E), and CovidRetro collaboration (see supplementary material F) were responsible for primary data collection. RZR, DF, MM, PH, RKG, RN, PN, MN, and ML were responsible for the primary data analysis. RZR and SKK were responsible for the meta-analysis. VdJ and RZR were responsible for the sensitivity analysis. VDJ and RZR were responsible for the initial draft of the manuscript. TD, TT, TLN, ML, FWA, LM, JD, BVC, LW, and KM revised the initial draft. RZR was responsible for the supplementary material on data and results (supplementary material A and D). VdJ and TT were responsible for the supplementary material on models (B). All authors contributed to the critical revision of the manuscript, approved the final version of the manuscript and agree to be accountable for the content. VdJ and RZR contributed equally. VdJ, TD, and KM are the guarantors of this manuscript.

  • Funding: This project received funding from the European Union’s Horizon 2020 research and innovation programme under ReCoDID grant agreement No 825746. This research was supported by the National Institute for Health and Care Research (NIHR) Leicester Biomedical Research Centre. RKG is supported by the NIHR. MN is supported by the Wellcome Trust (207511/Z/17/Z) and by NIHR Biomedical Research Funding to University College London and University College London Hospital. MM is supported by ELIXIR CZ research infrastructure project (MEYS grant No LM2018131), including access to computing and storage facilities. The CAPACITY-COVID registry is supported by the Dutch Heart Foundation (2020B006 CAPACITY), ZonMw (DEFENCE 10430102110006), the EuroQol Research Foundation, Novartis Global, Sanofi Genzyme Europe, Novo Nordisk Nederland, Servier Nederland, and Daiichi Sankyo Nederland. The Dutch Network for Cardiovascular Research, a partner within the CAPACITY-COVID consortium, received funding from the Dutch Heart Foundation (2020B006 CAPACITY) for site management and logistic support in the Netherlands. ML is supported by the Alexandre Suerman Stipend of the University Medical Centre Utrecht. FWA is supported by CardioVasculair Onderzoek Nederland 2015-12 eDETECT and by NIHR University College London Hospital Biomedical Research Centre. LW and BVC are supported by the COPREDICT grant from the University Hospitals KU Leuven, and by Internal Funds KU Leuven (C24M/20/064). The funders had no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the article for publication in the analysis and interpretation of data, in the writing of the report, and in the decision to submit the article for publication. We operated independently from the funders.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at https://www.icmje.org/disclosure-of-interest/ and declare: funding from the European Union’s Horizon 2020 research and innovation programme. ML and FWA have received grants from the Dutch Heart Foundation and ZonMw; FWA has received grants from Novartis Global, Sanofi Genzyme Europe, EuroQol Research Foundation, Novo Nordisk Nederland, Servier Nederland, and Daiichi Sankyo Nederland, and MM has received grants from Czech Ministry of Education, Youth and Sports for the submitted work; RKG has received grants from National Institute for Health and Care Research; FS has received an AWS DDI grant and grants from University of Sheffield and DBCLS; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; TD works with International Societiy for Pharmacoepidemiology Comparative Effectiveness Research Special Interest Group (ISPE CER SIG) on methodological topics related to covid-19 (non-financial); no other relationships or activities that could appear to have influenced the submitted work.

  • The manuscript’s guarantors (VdJ, TD, and KM) affirm that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as originally planned (and, if relevant, registered) have been explained. All authors had access to statistical reports and tables. Authors did not have access to all data, for privacy, ethical and/or legal reasons. Authors listed under “Primary data collection” in the contributorship section had access to data and take responsibility for the integrity of the data. Authors listed under the analysis bullets in the contributorship section take responsibility for the accuracy of the respective data analyses. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Dissemination to participants and related patient and public communities: We plan to share the results of this study on multiple social media platforms, including Twitter and LinkedIn. Copies of the manuscript will be sent to contributing centres, as well as being shared on the ReCoDID (www.recodid.eu) and COVID-PRECISE (www.covprecise.org) websites.

Data availability statement

The data from Tongji Hospital, China that support the findings of this study are available from https://github.com/HAIRLAB/Pre_Surv_COVID_19. Data collected within CAPACITY-COVID is available on reasonable request (see https://capacity-covid.eu/for-professionals/). Data for the CovidRetro study are available on request from MM or the secretariat of the Institute of Microbiology of the Czech Academy of Sciences (contact via mbu@biomed.cas.cz) for researchers who meet the criteria for access to confidential data. The data are not publicly available owing to privacy restrictions imposed by the ethical committee of General University Hospital in Prague and the GDPR regulation of the European Union. We can arrange to run any analytical code locally and share the results, provided the code and the results do not reveal personal information. The remaining data that support the findings of this study are not publicly available.

http://creativecommons.org/licenses/by/4.0/

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

View Full Text
Back to top