Intended for healthcare professionals

  1. Research
  2. ROBINS-I: a tool for...
  3. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
CCBYNC Open access
Research Methods & Reporting

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

BMJ 2016; 355 doi: https://doi.org/10.1136/bmj.i4919 (Published 12 October 2016) Cite this as: BMJ 2016;355:i4919
  1. Jonathan AC Sterne, professor1,
  2. Miguel A Hernán, professor2,
  3. Barnaby C Reeves, professorial research fellow3,
  4. Jelena Savović, research fellow1 4,
  5. Nancy D Berkman, senior health policy research analyst5,
  6. Meera Viswanathan, director6,
  7. David Henry, professor7,
  8. Douglas G Altman, professor8,
  9. Mohammed T Ansari, adjunct professor9,
  10. Isabelle Boutron, professor10,
  11. James R Carpenter, professor11,
  12. An-Wen Chan, Phelan scientist12,
  13. Rachel Churchill, professor13,
  14. Jonathan J Deeks, professor14,
  15. Asbjørn Hróbjartsson, professor15,
  16. Jamie Kirkham, lecturer16,
  17. Peter Jüni, professor17,
  18. Yoon K Loke, professor18,
  19. Theresa D Pigott, professor19,
  20. Craig R Ramsay, professor20,
  21. Deborah Regidor, senior consultant21,
  22. Hannah R Rothstein, professor22,
  23. Lakhbir Sandhu, resident23,
  24. Pasqualina L Santaguida, assistant professor24,
  25. Holger J Schünemann, professor25,
  26. Beverly Shea26,
  27. Ian Shrier, investigator27,
  28. Peter Tugwell, professor28,
  29. Lucy Turner, senior research associate29,
  30. Jeffrey C Valentine, associate professor30,
  31. Hugh Waddington, lecturer31,
  32. Elizabeth Waters, professor (deceased 2015)32,
  33. George A Wells, professor33,
  34. Penny F Whiting, senior research fellow34,
  35. Julian PT Higgins, professor35
  1. 1School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK
  2. 2Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA; and Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, Massachusetts, USA; and Harvard-Massachusetts Institute of Technology Division of Health Sciences and Technology, Boston, Massachusetts, USA
  3. 3School of Clinical Sciences, University of Bristol, Bristol, BS2 8HW, UK
  4. 4National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, Bristol BS1 2NT, UK
  5. 5Program on Health Care Quality and Outcomes, Division of Health Services and Social Policy Research, RTI International, Research Triangle Park, NC 27709, USA
  6. 6RTI-UNC Evidence-based Practice Center, RTI International, Research Triangle Park, NC 27709, USA
  7. 7Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
  8. 8Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology & Musculoskeletal Sciences, University of Oxford, Oxford OX3 7LD, UK
  9. 9School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada
  10. 10METHODS Team, Centre of Epidemiology and Statistics Sorbonne Paris Cité Research, INSERM UMR 1153, University Paris Descartes, Paris, France
  11. 11Department of Medical Statistics, London School of Hygiene and Tropical Medicine and MRC Clinical Trials Unit at UCL, London, UK
  12. 12Women's College Research Institute, Department of Medicine, University of Toronto, Canada
  13. 13Centre for Reviews and Dissemination, University of York, York, YO10 5DD, UK
  14. 14Institute of Applied Health Research, University of Birmingham, Birmingham B15 2TT, UK
  15. 15Center for Evidence-Based Medicine, University of Southern Denmark & Odense University Hospital, 5000 Odense C, Denmark
  16. 16Department of Biostatistics, University of Liverpool, Liverpool, L69 3GL, UK
  17. 17Applied Health Research Centre (AHRC), Li Ka Shing Knowledge Institute of St Michael’s Hospital, and Department of Medicine, University of Toronto, Toronto, Ontario, Canada
  18. 18Norwich Medical School, University of East Anglia, Norwich NR4 7TJ, UK
  19. 19School of Education, Loyola University Chicago, Chicago, IL 60611, USA
  20. 20Health Services Research Unit, University of Aberdeen, Aberdeen, AB25 2ZD, UK.
  21. 21Evidence Services, Kaiser Permanente, Care Management Institute, Oakland, CA 94612, USA
  22. 22Department of Management, Zicklin School of Business, Baruch College—CUNY, New York, NY 10010, USA
  23. 23Division of General Surgery, University of Toronto, Toronto, Canada
  24. 24Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, L8S 4K1, Canada
  25. 25Departments of Clinical Epidemiology and Biostatistics and of Medicine, Cochrane Applicability and Recommendations Methods (GRADEing) Group, MacGRADE center, Ontario, L8N 4K1, Canada
  26. 26Ottawa Hospital Research Institute, Center for Practice Changing Research and School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada
  27. 27Centre for Clinical Epidemiology, Lady Davis Institute, Jewish General Hospital, McGill University, Montreal, Quebec, Canada
  28. 28Department of Medicine and School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, Ottawa, Ontario, Canada
  29. 29Ottawa Hospital Research Institute, Ottawa, ON, Canada
  30. 30University of Louisville, Louisville, KY 40292, USA
  31. 31International Initiative for Impact Evaluation, London School of Hygiene and Tropical Medicine, and London International Development Centre, London, UK
  32. 32Jack Brockhoff Child Health & Wellbeing Program, Melbourne School of Population and Global Health, University of Melbourne, Melbourne, VIC 3010, Australia
  33. 33School of Epidemiology, Public Health and Preventive Medicine and Director, Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, Ottawa, Ontario, K1Y 4W7, Canada
  34. 34School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK; and National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West (NIHR CLAHRC West) at University Hospitals Bristol NHS Foundation Trust, Bristol BS1 2NT, UK
  35. 35School of Social and Community Medicine, University of Bristol, Bristol BS8 2PS, UK
  1. Correspondence to: J A C Sterne jonathan.sterne{at}bristol.ac.uk

Non-randomised studies of the effects of interventions are critical to many areas of healthcare evaluation, but their results may be biased. It is therefore important to understand and appraise their strengths and weaknesses. We developed ROBINS-I (“Risk Of Bias In Non-randomised Studies - of Interventions”), a new tool for evaluating risk of bias in estimates of the comparative effectiveness (harm or benefit) of interventions from studies that did not use randomisation to allocate units (individuals or clusters of individuals) to comparison groups. The tool will be particularly useful to those undertaking systematic reviews that include non-randomised studies.

Summary points

  • Non-randomised studies of the effects of interventions are critical to many areas of healthcare evaluation but are subject to confounding and a range of other potential biases

  • We developed, piloted, and refined a new tool, ROBINS-I, to assess “Risk Of Bias In Non-randomised Studies - of Interventions”

  • The tool views each study as an attempt to emulate (mimic) a hypothetical pragmatic randomised trial, and covers seven distinct domains through which bias might be introduced

  • We use “signalling questions” to help users of ROBINS-I to judge risk of bias within each domain

  • The judgements within each domain carry forward to an overall risk of bias judgement across bias domains for the outcome being assessed

Non-randomised studies of the effects of interventions (NRSI) are critical to many areas of healthcare evaluation. Designs of NRSI that can be used to evaluate the effects of interventions include observational studies such as cohort studies and case-control studies in which intervention groups are allocated during the course of usual treatment decisions, and quasi-randomised studies in which the method of allocation falls short of full randomisation. Non-randomised studies can provide evidence additional to that available from randomised trials about long term outcomes, rare events, adverse effects and populations that are …

View Full Text
Back to top