RoB 2: a revised tool for assessing risk of bias in randomised trials
BMJ 2019; 366 doi: https://doi.org/10.1136/bmj.l4898 (Published 28 August 2019) Cite this as: BMJ 2019;366:l4898
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Sterne et al. present RoB2, a revised tool for assessing risk of bias in randomized trials. The finalized version is rather complex. I would like to propose three changes that might help to clarify the new tool.
It seems that RoB2 makes it easier for trial results to be rated as low risk of bias for a particular domain although the overall risk of bias corresponds to the worst risk of bias in any of the domains. This delicate balance might be lost if reviewers are allowed to downplay the overall risk of bias. I would, therefore, recommend adding a requirement to use the overall risk of bias judgment in interpreting results and formulating a conclusion. Otherwise, RoB2 would simply lower the bar and rate trial results at a lower risk of bias than the previous assessment tool. [1] Without a focus on the overall risk of bias, the traffic light presentation would show more green lights than was previously the case. This could be misleading to readers unaware of how the overall judgment should be reached.
Secondly, question 4.5 seems redundant. After asking if outcome assessors were aware of the intervention received and if that knowledge could have influenced the assessment of the outcome, question 4.5 asks whether this was likely or not. The elaboration in the supplementary material does not give an example where such influence is not likely and it’s rather difficult to imagine a relevant scenario. A trial in which patients are not blinded and subjective outcomes are used, for example, is at high risk of bias. [2] Low expectations measured before the trial would change little to that, as patients’ hopes and expectations might alter during the treatment or following contact with (unblinded) therapists. As question 4.5 adds little to the risk of bias assessment and could easily be misused, I would recommend removing it from RoB2. Answers ‘yes’, ‘probably yes’, and ‘no information’ to question 4.4 would then directly result in a high risk of bias judgment.
Finally, there seems to be an issue with question 5.1 which asks if the analysis of the results were in accordance with a pre-specified analysis plan. The elaboration states that “changes to analysis plans that were made before unblinded outcome data were available (…) do not raise concerns about bias in selection of the reported result.” I suspect that in open-label trials, researchers can get a hint of the size and direction of the results before looking at the data. The previous Cochrane risk of bias tool required all relevant outcomes to be reported in a pre-specified way for a study to be rated as low risk of bias if a protocol was available. [1] I would recommend restoring this requirement as it is less ambiguous and failure to meet it would only result in ‘some concerns’, not a high risk of bias judgment.
References
[1] Higgins JPT, Altman DG, Gøtzsche PC, et al, Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. doi:10.1136/bmj.d5928
[2] Savović J, Jones H, Altman D, Harris R, Jűni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012 Sep;16(35):1-82. doi: 10.3310/hta16350
Competing interests: No competing interests
In the recent publication in BMJ of the finalised Risk of Bias 2 tool (RoB2), used in the context of the GRADE system by Cochrane for assessing quality of clinical trial evidence, the discussion claims that changes made are likely to reduce stringency of RoB assessment, especially in unblinded trials. This looks to be a retrograde step, since unblinded trials with subjective outcomes appear increasingly to be given more credence than deserved. What is also of concern is that the corresponding author, Jonathan Sterne, is an author on the report of the unblinded SMILE trial of the Lightning Process for myalgic encephalomyelitis/ chronic fatigue syndrome (ME/CFS) (Crawley et al. 2018). This report has been severely criticised for multiple methodological errors. Despite rewriting, it is still considered inadequate by over 50 academics and clinicians (Tuller, 2019).
The proposals the RoB2 document makes include handling of problems with bias due to beliefs held by the patient or treatment delivery team when outcome measures are subjective and also the changing of outcome measures after trial initiation but before data analysis. Problems with the SMILE trial include both severe risk of bias relating to subjective outcomes and outcome switching midstream.
The GRADE system takes the premise that randomised trials provide high quality evidence unless they suffer from one or more defects, including bias. The fragility of this can be illustrated by proposing a trial that randomises patients to being taught to think and say they feel better, whether they do or not, or to being told to say how they really feel, using how the patient says they feel as outcome. One might think such a trial would never be proposed. Yet this appears to be more or less what the SMILE trial was (at least from the little we know about the Lightning Process). Moreover, the report does not tell us much, and it looks as if according to RoB2 the risk of bias is not scored high if we do not know enough to have specific reasons for doing so!
RoB2 does mention the possibility that bias might arise from patient or therapist beliefs, citing a physiotherapist assessing her own treatment or a patient reporting their response to homeopathy, as if to indicate bias is restricted to such self-evident examples. There appears to be no recognition that expectation bias due to beliefs about the outcome of an intervention is ubiquitous in trials (and any experiment, including laboratory work). In ME/CFS we have seen major responses due to expectation bias with conventional drugs now known to have no therapeutic effect, such as rituximab. The RoB2 analysis seems either naïve or disingenuous.
The SMILE trial had a highly unsatisfactory structure, starting as a ‘feasibility study’, recruiting more than half of the patients, and then being registered after switching of outcomes in the knowledge of progress of early participants. This might fall under RoB2’s condition of before data analysis but is a classic cherry-picking scenario. ‘Feasibility’ and ‘pragmatic’ trials appear increasingly popular. These terms look like attempts to legitimise methods that break basic rules for gathering reliable evidence. It needs to be acknowledged that ‘methodological experts’ associated with clinical trials units and related departments may have a conflict of interest in terms of co-authorship on publications of such trials.
We appear to be moving towards acceptance of methodology that for decades we have known yields meaningless results. I am concerned that bodies like Cochrane and the BMJ are sleepwalking into a situation where they rubber stamp commercial ventures of no merit.
References
Crawley, E.M., Gaunt, D.M., Garfield, K., et al. (2018). Arch. Dis. Childhood 103,2. https://adc.bmj.com/content/103/2/155
Tuller, D (2019). http://www.virology.ws/2019/08/28/trial-by-error-an-open-letter-to-dr-go...
Competing interests: No competing interests
Re: RoB 2: a revised tool for assessing risk of bias in randomised trials
At the end of 2013 there was a debate held at the Cochrane Methods Symposium in Quebec on whether funding source should be a part of the Cochrane Risk of Bias tool. Lisa Bero argued in favour [1] and Jonathan Sterne against [2]. Sterne’s argument concluded that to add funding source as a standard item would send a negative message to industry and would stop collaborative work to help address problems with industry research. I argue that it is not only industry research that has serious problems with bias which need to be addressed. Much greater attention should be paid to the way the self-interest of researchers both inside and outside industry can bias published research findings.
Triallists from both industry and academia often have financial success and reputation invested in a favourable result. These conflicts will affect many decisions including the choice of outcomes, and outcome measures, and are likely to flatter a favoured intervention. If things still don’t seem to be going the right way, triallists are given considerable leeway to make post-hoc adjustments, particularly if they are eminent clinicians and/or experienced triallists from prestigious institutions.
Both the old and revised version of the RoB tool focus on identifying bias using information published in journal articles, and by liaising with triallists. The bias affecting decisions taken before data collection is not considered. There is also no systematic assessment of bias affecting review authors’ decisions when conducting a review.
An important part of systematic reviewers’ role is to expose biased research practices to ensure patients are properly informed of any potential problems which lead to endorsement of treatments that could be ineffective or harmful. For example, trials of interventions such as exercise, where blinding is not possible, are unlikely to yield useful data unless both self-report and objective outcome measures are used [3]. Bias caused by triallists’ decisions to rely on subjective outcomes in unblinded trials is compounded if systematic reviewers follow suit. RoB 2 seems likely to do a worse job of identifying poor primary research practices, such as relying on subjective outcomes in unblinded trials.
The developers of RoB 2 have an explicitly stated expectation that the refinements to the tool will lead to a greater proportion of trial results being assessed as at low risk of bias. Increased leeway for judging trials at low risk of bias will simply serve to obscure the serious problems which affect the reliability of trial data. Individual outcomes can now be rated as having a low risk of selective reporting bias if protocol deviations occurred prior to data unblinding. There is no provision made for the fact that in unblinded trials, early indications of results can easily be discerned. Considering the problems afflicting important areas of medical research, this is a backward step in Cochrane’s stated mission to identify and summarise only the most reliable research evidence. It seems likely that the use of RoB 2 will further weaken the hand of those wishing to challenge over-optimistic evaluations of the evidence by recommending reviewers ignore important indications that the conduct of included studies was influenced by the pursuit of a positive result.
Going back to Bero’s argument that reviews should include information about funding sources, I would go further. I think it would be useful to systematically collect and present funding sources and additional observed factors about included studies. Information could include funding source, researcher allegiance (involvement in development and/or use of intervention in their clinical practice), balance of self-report and objective measures in unblinded trials, and level of patient involvement (in choice of outcomes, etc.). This would give readers a more complete overview of previous research including both positive (e.g. patient involvement) and negative factors (e.g. inappropriate outcome choice) affecting the risk of bias. It could help illustrate how conflicts of interest, both financial and reputational, may lead to serious bias in favour of review findings which help bolster academic and clinical careers rather than protect patients from ineffective or harmful treatments.
[1] Bero LA. Why the Cochrane Risk of Bias Tool Should Include Funding Source as a Standard Item. Cochrane Database of Systematic Reviews 2013, Issue 12. Art. No.: ED000075. DOI: 10.1002/14651858.ED000075.
[2] Sterne JAC. Why the Cochrane Risk of Bias Tool Should not Include Funding Source as a Standard Item. Cochrane Database of Systematic Reviews 2013, Issue 12. Art. No.: ED000076. DOI: 10.1002/14651858.ED000076
[3] Asbjørn Hróbjartsson, Frida Emanuelsson, Ann Sofia Skou Thomsen, Jørgen Hilden, Stig Brorson, Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind sub-studies, International Journal of Epidemiology, Volume 43, Issue 4, August 2014, Pages 1272–1283, https://doi.org/10.1093/ije/dyu115
Competing interests: No competing interests