Many reviews referred to as 'systematic reviews' in ecology are not consistent with best practice in that they generally lack appropriate critical appraisal of included studies. This limitation is particularly important in applied ecology, where there have been increasing calls for more systematic reviews to guide decision making. To identify the available critical appraisal tools (CATs) and hierarchies of evidence available for ecology studies, we systematically searched for: studies that described the development and/or examination of tools to assess the potential methodological bias in studies of ecology; and the tools used to assess potential methodological bias of included studies in ecological systematic reviews. We identified 680 reviews labelled as 'systematic reviews' in ecology, however only 4.0% performed critical appraisal of the included studies. Three hierarchies of evidence and 23 CATs were identified, and assessed as lacking independent development, validity and reliability testing, and/or completeness. The authors of the reviews that included critical appraisal have appropriately identified the need to move reviews in ecology in the direction of this higher level of evidence, and have taken applied ecology further in the direction of evidence-based practice. However, we identified shortcomings in these approaches when compared with best practice, and conclude that new tools are needed that reflect a range of questions posed in ecology. Through increasing the availability of such tools, the strength of evidence provided by systematic reviews in ecology would improve.