Criteria used in existing tools to inform a judgement of ‘high’ risk of bias due to selective publication
| ‘High’ risk of bias criteria proposed in existing tools | AHRQ RRB | GRADE | SAQAT | NMA-Quality | Total, n (%) |
| Assessment directed at a specific synthesis (eg, meta-analysis) | |||||
| Evidence of funnel plot asymmetry (based on visual inspection of funnel plot or statistical test for funnel plot asymmetry) | ✓ | ✓ | ✓ | ✓ | 4 (100) |
| Smaller studies tend to demonstrate more favourable results (based on visual assessment, without funnel plot) | ✓ | 1 (25) | |||
| Clinical decision would differ for estimates from a fixed-effect versus a random-effects model because the findings from a fixed-effect model are closer to the null | ✓ | 1 (25) | |||
| Substantial heterogeneity in the meta-analysis cannot be explained by some clinical or methodological factor | ✓ | 1 (25) | |||
| At least one study is affected by non-publication or non-accessibility | ✓ | 1 (25) | |||
| Presence of small (often ‘positive’) studies with for-profit interest in the synthesis | ✓ | ✓ | 2 (50) | ||
| Presence of early studies (ie, set of small, ‘positive’ trials addressing a novel therapy) in the synthesis | ✓ | ✓ | 2 (50) | ||
| Discrepancy in findings between published and unpublished trials | ✓ | ✓ | ✓ | 3 (75) | |
| Search strategies were not comprehensive | ✓ | ✓ | ✓ | 3 (75) | |
| Methods to identify all available evidence were not comprehensive | ✓ | ✓ | 2 (50) | ||
| Grey literature were not searched | ✓ | 1 (25) | |||
| Restrictions to study selection on the basis of language were applied | ✓ | 1 (25) | |||
| Industry influence may apply to studies included in the synthesis | ✓ | 1 (25) |
AHRQ RRB, AHRQ tool for evaluating the risk of reporting bias29; GRADE, GRADE rating of quality of evidence34–37; NMA-Quality, Framework for evaluating the quality of evidence from a network meta-analysis49; SAQAT, Semi-Automated Quality Assessment Tool.45 46