In the wake of an increasing political commitment to evidence-based decision making and evidence-based educational reform that emerged with the No Child Left Behind effort, the question of what counts as evidence has become increasingly important in the field of science education. In current public discussions, academics, politicians, and other stakeholders tend to privilege experimental studies and studies using statistics and large sample sizes. However, some science education studies use a lot of statistics and large sample sizes and yet, as the author suggests in this article, are flawed and do not provide (sound) evidence in favor of some treatment or claim. Leaving aside the assertion and consensus of researchers across the quantitative/qualitative spectrum, one must ask whether all studies that appear to provide "quantitative" support for a particular effect do in fact provide "substantial" or strong evidence. As an anonymous reviewer of this contribution has pointed out, the question in its title really has two dimensions: (1) What constitutes "valid" evidence; and (2) what are the limits of the claims that can be constructed when the evidentiary chain from premises to results is perfectly constructed. Both are important in constructing explanations for phenomena of interest to scientists generally and to science educators in particular. The author begins by discussing the two issues in the context of the logic of scientific inquiry and statistical inference and then exemplifies the issues as these play out in one recent article published in the pages of this journal (Bodzin, 2011). To further concretize his discussion, the author also sketches two re-analyses concerned with the weight of the evidence provided by: (1) 10 studies of paranormal psychological phenomena ("psi"); and (2) 855 studies in experimental psychology. (Contains 1 table, 1 figure, and 8 notes.)