One purpose of education research is to develop and rigorously evaluate the effectiveness of programs for supporting students' learning and achievement. The Institute of Education Sciences has amplified that purpose (Shadish & Cook, 2009) and attempted to improve the methodological standards for conducting such work--primarily through the What Works Clearinghouse (WWC), which, since 2002, has supported an ongoing effort to synthesize research on the effectiveness of educational interventions, programs, and policies. According to its stringent, methodological standards, only "well-designed and well-implemented" randomized controlled trials and studies employing quasi-experimental designs with equating or matching are included in the WWC's 15 topical syntheses, one of which is mathematics. Despite repeated calls for increased attention to theory in program evaluation for more than 25 years, recent reviews of the literature suggest that the field has seen little change (Confrey & Stohl, 2004; Coryn, Noakes, Westine, & Schröter, 2010; Weiss, 1997). The requirement that evaluators link research designs to the theories underlying the programs they evaluate is not included in the WWC's standards for rigorous evaluations. Given the considerable differences in mathematical goals and theories of learning from which mathematics programs are designed, and the high stakes for students' mathematics learning and academic futures (as well as for the fortunes of program developers), it is important to ask whether the WWC's methodological specifications are sufficient, or whether program evaluation has once again lost (or perhaps never gained) sight of the role of theory in evaluation design and implementation (Bickman, 1987), resulting in overly-constrained and uninterpretable syntheses of otherwise methodologically strong evaluation research (Schoenfeld, 2006). The purpose of this paper is to determine whether calls for theory-based evaluation research have had an impact on the extent to which evaluators of mathematics programs attend to program theory in their design, implementation, and reporting of studies. For each report that met stringent methodological standards (i.e., WWC evidence standards), the authors asked: (1) What type of program theory was articulated (none, sub-theoretical, or theoretical; Lipsey et al., 1985); (2) What was the quality of evaluators' articulation of program theory (entirely implicit, drawn from a limited number of resources such as developer or publisher descriptions, or drawn from multiple resources and situated in the research literature); (3) How was the articulated program theory used in the evaluation; and (4) To what extent does attention to program theory vary by characteristics of the program and evaluator(s), such as the nature of the program's mathematics learning goals and instruction (i.e., back-to-basics, typical, inquiry-based, or blended); type of publication (e.g., peer-reviewed journal or technical report); funding source (e.g., Federal/state agency or publisher/developer); and timing of analysis (i.e., primary or secondary analysis)? Modeled after the WWC's process, every report was examined and coded independently by two individuals employing the same coding scheme, and research questions listed here. The findings of evaluation studies guide the decisions of policy makers at every level, including the adoption of both curriculum materials and intervention programs. These decisions are consequential for students' mathematics learning and academic futures. It is therefore crucial that evaluators "get it right" when assessing the effectiveness of such programs. This analysis indicates that WWC's methodological specifications are inadequate because they overlook understanding and using theory in evaluation design and implementation. In general, evaluation research of mathematics programs needs to improve in its attention to and use of program theory. Tables and figures are appended.