AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both

We are very grateful to Mr. Long Ge, Assoc Prof Jin-hui Tian and Prof Ke-hu Yang for their interest in AMSTAR 2 and for taking the time and effort to provide feedback. Our responses to their helpful comments are below. We have reproduced their comments and our replies are marked *

Item 2 (a protocol prior to conduct), it’s extremely important to develop a protocol of systematic review in advance. AMSTAR 2 answers thS reviews reported a protocol [2]. When a protocol of systematic review doesn’t exist, “Not applicable” or “No protocol available” should be considered.
* We agree it is important to have a written protocol before commencing a systematic review. To achieve a partial Yes from AMSTAR 2 authors are required to state that they had a written protocol or guide that included  review question(s)  a search strategy  inclusion/exclusion criteria  a risk of bias assessment. For a full Yes the protocol should be registered. We understand the writers’ concerns, but do not think this is an excessively demanding standard as the review team should have considered these steps and written them down. We don’t believe there will be many situations where such a requirement is ‘not applicable’, and if no protocol is available it should be viewed as a negative and graded accordingly. Looking ahead it is possible that AMSTAR 2 will be used as a checklist for conducting reviews and authors will be aware that their review will be viewed as having a critical weakness if this criterion is not met.

Item 4 (Literature search strategy), AMSTAR 2 has considered all key contents except for the reliability of the literature search. Methodological studies and guidelines have highlighted the importance of searchers, involving a local healthcare librarian or information specialist could effectively avoid errors during the conduct of the search [3-5]. Therefore, it’s important to determine the reliability of search by identifying whether systematic reviewers involve an information specialist or whether the search strategies were peer-reviewed.
* We agree completely that it is desirable to have an information specialist involved in the literature search. We do not think that their absence invalidates an otherwise well-conducted search. In the manuscript/user guide we have recommended that review teams working with observational studies of intervention effects should include methods and content experts. However, we have not suggested that their absence is automatically a critical flaw. We think that review teams should consider the range of expertise needed for each review, which may vary substantially.

Item 7 (provide a list of excluded studies and justify the exclusions), this item should contain two parts, one is a list of excluded studies, and another is the reasons for exclusion. Our experience tell us most published non-Cochrane reviews don’t provide the list of excluded studies, but give a reason of exclusion and (or) a c. When systematic reviews just report the reason of exclusion and (or) a PRISMA flow graph, should evidence users answer this item as “Partial Yes” or “No”?
* AMSTAR 2 rates a response as partial Yes if the list of excluded (potentially relevant) studies is provided and a full Yes if the reason for exclusion of each of these studies is provided. We believe the list of excluded studies is the most important factor as it is possible to determine the effect of exclusions on the results of the review. So, in answer to the last point, providing only reasons for exclusion would be rated as No.

Item 11 (appropriate methods for statistical combination), AMSTAR 2 indicates that authors should report pooled estimates separately for the different study types. In fact, some systematic reviews including both randomised and non-randomised studies combine the estimates from two study types. And there are some methodological studies to support the combination of evidence from randomised and non-randomised studies [6,7]. However, it’s more important to justify the combination methods for different study types.
* In the manuscript and user guide we caution about the possible impact of combining the results of randomised and non-randomised studies of treatment effects, but did not state that should never be combined. We recommend that separate estimates of intervention effects from the two study types should be provided so that users of the review can determine if there are important differences between the two sets of analyses. We agree that justification for combining different study types is desirable

Item 15 (investigation of publication bias), quantitative synthesis is performed, authors should investigate the likelihood and magnitude of publication bias. “Yes” are answered when performing graphical or statistical tests for publication bias. However, the Cochrane Handbook doesn’t recommend performing graphical or statistical tests for publication bias when the number of included studies is less than 10. In this situation, how do we assess this item?
* In the user guide we state: “The key issues are whether the authors have done their best to identify PB through deeper and intensive literature searches (as needed and according to the setting), shown an awareness of the likely impact of PB in their interpretation and discussion of the results and performed a sensitivity analyses to determine how many missing ‘null’ studies would be needed to invalidate the results they obtained.”

Once again, we thank the writers for their careful reading of the manuscript and for providing these comments, which we hope are helpful to other users of the instrument

Beverley Shea, Senior Methodologist and Clinical Investigator, Ottawa Hospital Research Institute, Ottawa, Canada

David Henry, Professor, Centre for Research in Evidence-Based Practice, Bond University, Gold Coast, Australia