ERIC Number: ED656933
Record Type: Non-Journal
Publication Date: 2021-Sep-28
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Has Godot Arrived? An Update on the State of Economic Evaluation in Education Research
Rob Shand; A. Brooks Bowden; Rebecca Davis
Society for Research on Educational Effectiveness
Background: The salience of economic evaluation in education -- cost analysis, cost-effectiveness analysis, and benefit-cost analysis -- has been on the rise in recent years. In contexts when resources are especially scarce, research to support decision-making to maximize benefits to students is especially valuable. Economic evaluation can help to provide evidence on the strategic and effective ways to address both new and persistent challenges, such as those presented by COVID-19. Nonetheless, researchers have bemoaned the relative paucity of research on the costs of educational interventions since the method was formalized (Levin, 1975). In fact, as of 2002, the vast majority of studies purporting to analyze costs or cost-effectiveness in education were primarily rhetorical in nature (Clune, 2002). Levin (2001) attributed this to a lack of demand for rigorous cost studies among decision-makers and a lack of training. Purpose: Cost studies have garnered new attention in recent years with their adoption by institutions such as JPAL (Dhaliwal, Duflo, Glennerster & Tulloch, 2013) and as a required component for evaluations funded by the Institute of Education Sciences. Given this new attention, an update on the prevalence and quality of cost studies in education is in order. We seek to address the following research questions on the state of the literature on economic evaluation in education: Has the prevalence of economic evaluation increased in education research in light of its increased salience? Has the quality and rigor of economic evaluation increased in recent years? Does the prevalence and quality of studies vary by type of analysis and area of specialization? Do rigorous, high-quality studies share common features that are relevant for future researchers and for decision-makers? Do economic evaluations impact education policy in a documentable way? Setting & Subjects: We include in our analysis any education paper that appeared in our search (described below) since 2007 that includes a cost component, even if the cost component was secondary to the main analysis. We include all levels of education and all articles published after 2007 to pick up where the most recent comprehensive review of the literature left off (Ross, Barkoui & Scott, 2007). We include papers regardless of country of origin as long as the paper was available in English. Research design, data collection, & analysis: Our initial list of articles was drawn from a search on the terms "cost," "cost-effectiveness," "cost-benefit," and "economic evaluation" were searched alongside "education" using Web of Science, Google Scholar, EconLit, and ERIC. We found nearly 4,000 citations, the vast majority only tangentially related to education and thus ineligible for analysis. We supplemented this list with additional articles that cited the second and third editions of the Economic Evaluation in Education textbook (Levin & McEwan, 2001 & Levin, McEwan, Bel eld, Bowden & Shand, 2018). The textbook citations list was subjected to the same inclusion criteria as the web search. This work builds on prior comprehensive reviews of the state of economic evaluation of education by adding a novel look at policy reach and influence. We record a list of policy documents that cite each paper, a count of citations, and an Almetric figure. We used the Dimensions and Google Scholar search engines to record each article's citation count and used Dimensions to record the article's presence in policy documents along with its Almetric score. We coded original empirical cost data or specific recommendations for cost analysis on a numeric rubric for quality and rigor, based on the economic evaluation checklist presented in Levin et al. 2018. Papers received a rigor score based on their adherence to the rubric, with papers that met none of these criteria coded 0. Finally, we conducted descriptive analyses on the prevalence and quality of cost studies in education: the number of studies by year, type, area, and quality, and tests to determine if there are statistically significant differences in quality by type and topic of analysis. Findings: Fifty-one percent of the 104 papers analyzed to date were "real" in the sense that they had original cost data or specific recommendations on cost analysis, and 23% of the papers were both authentic cost analyses and rated as high-quality on our rubric. This represents significant progress from the Clune (2002) study, when only 5% of papers met these criteria, although significant work remains to reach even a majority of high-quality cost-studies. Cost-effectiveness analysis is by far the most common type of analysis and the preponderance of papers have been published in higher education, social and emotional learning, general K-12 reforms, and teachers. Quality seems dispersed across topic areas, with most papers coded as partly meeting rigor criteria and no statistically significant differences from the expected distribution based on chi-squared and Fisher's exact tests. Our analysis of policy reach showed a range of influence. Several of the papers included in our analysis had outsized influence but the bulk of papers had modest influence. Conclusions: This review of the economic evaluation literature shows some significant progress since previous reviews. There does indeed seem to be an uptick in both number and quality of studies, notably among cost-effectiveness analyses and in the areas of higher education and social and emotional learning. Nonetheless, there is still much work to be done, as original empirical cost analyses remain relatively rare. Even among this sample, half of the papers had little or no original empirical analysis, suggesting that gathering cost data and performing rigorous analysis remain obstacles. While the increase in frequency and quality is encouraging, further uptake of high-quality cost evaluations would help to fill an essential need in education. In considering the classic tradeoff in economics between efficiency and equity, Rice (2004) argued that applications of economic evaluation to education research can help resolve the dichotomy and simultaneously address both goals in a number of ways. A focus on how resources contribute to outcomes can help provide a more nuanced understanding of the relationship between inputs, processes, and outcomes, moving beyond debates on whether money matters to focus on how to use resources to maximize the public good.
Descriptors: Educational Research, Cost Effectiveness, Intervention, Costs, Economics, Educational Quality, Higher Education, Educational Policy
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Information Analyses
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A