Table 5

Criteria used in existing tools to inform a judgement of ‘high’ risk of bias in selection of the reported result

‘High’ risk of bias criteria proposed in existing toolsAHRQ ORBDowns-BlackRoB 1.0RoB 2.0SYRCLE RoBRoBANSReidROBINS-ITotal, n (%)
Assessment directed at study as a whole
 One or more reported outcomes were not prespecified (unless clear justification for their reporting is provided, such as an unexpected adverse event)✓✓✓✓4 (50)
 One or more outcomes were reported using measurements, analysis methods or subsets of the data (eg, subscales) that were not prespecified✓✓2 (15)
 One or more retrospective, unplanned, subgroup analyses were reported✓1 (13)
 Any analyses that had not been planned at the outset of the study were not clearly indicated✓1 (13)
Assessment directed at a specific outcome/result
 Particular outcome was not prespecified but results were reported✓1 (13)
 Reported result for a particular outcome is likely to have been selected, on the basis of the findings, from multiple outcome measurements (eg, scales, definitions, time points) within the outcome domain✓✓2 (25)
 Reported result for a particular outcome is likely to have been selected, on the basis of the findings, from multiple analyses of the data✓✓2 (25)
 Reported result for a particular outcome is likely to have been selected, on the basis of the findings, from different subgroups✓1 (13)

  • AHRQ ORB, AHRQ outcome and analysis reporting bias framework28; Downs-Black, Downs Black tool31; Reid, Reid et al selective reporting bias algorithm47; RoB 1.0, Cochrane risk of bias tool for randomised trials21 39 40; RoB 2.0, Revised tool for assessing risk of bias in randomised trials41 42; RoBANS, Risk of Bias Assessment Tool for Nonrandomized Studies44; ROBINS-I, Risk Of Bias In Non-randomized Studies of Interventions tool4; SYRCLE RoB, SYstematic Review Centre for Laboratory animal Experimentation risk of bias tool.43