Conducting research with human subjects can be difficult because of limited sample sizes and small empirical effects. We demonstrate that this problem can yield patterns of results that are practically indistinguishable from flipping a coin to determine the direction of treatment effects. We use this idea of random conclusions to establish a…

The purpose of this study is to present a set of empirically derived effect size distributions in order to provide field-based benchmarks for assessing the relative effects of interventions aimed at reducing challenging behavior or increasing engagement for young children with and without disabilities. We synthesized 192 single-case designs that…

Randomized experiments are considered the gold standard for causal inference because they can provide unbiased estimates of treatment effects for the experimental participants. However, researchers and policymakers are often interested in using a specific experiment to inform decisions about other target populations. In education research,…

Purpose: The purpose of this study was to document, validate, and corroborate effect size (ES) for single­-subject design in treatment of children with functional phonological disorders; to evaluate potential child-­specific contributing variables relative to ES; and to establish benchmarks for interpretation of ES for the population. Method: Data…

In education research, small samples are common because of financial limitations, logistical challenges, or exploratory studies. With small samples, statistical principles on which researchers rely do not hold, leading to trust issues with model estimates and possible replication issues when scaling up. Researchers are generally aware of such…

This study examined how specific guidelines and heuristics have been used to identify methodological rigor associated with single-case research designs based on quality indicators developed by Horner et al. Specifically, this article describes how literature reviews have applied Horner et al.'s quality indicators and evidence-based criteria.…

Evaluators must employ research designs that generate compelling evidence related to the worth or value of programs, of which assessment data often play a critical role. This article focuses on assessment design in the context of evaluation. It describes the process of using the Framework for K-12 Science Education and Next Generation Science…

This article explores transnational activism within Education for All (EFA), looking specifically at the strategic use of information and research by transnational advocacy organizations. Through a comparative case-study examination of two prominent civil society organizations within the EFA movement--the Asia South Pacific Association for Basic…

In this article, we test an alternative approach to creating a counterfactual basis for estimating individual and average treatment effects. Instead of using control/comparison groups or before-measures, the so-called Counterfactual as Self-Estimated by Program Participants (CSEPP) relies on program participants' self-estimations of their own…

Optimal levels of treatment fidelity, a critical moderator of intervention effectiveness, are often difficult to sustain in applied settings. It is unknown whether performance feedback, a widely researched method for increasing educators' treatment fidelity, is an evidence-based practice. The purpose of this review was to evaluate the current…

The sharp regression discontinuity design (RDD) has three key weaknesses compared to the randomized clinical trial (RCT). It has lower statistical power, it is more dependent on statistical modeling assumptions, and its treatment effect estimates are limited to the narrow subpopulation of cases immediately around the cutoff, which is rarely of…

Interventions based on the results of functional behavioral assessment (FBA) have been the topic of extensive research and, in certain cases, mandated for students with disabilities under the Individuals With Disabilities Education Act. There exist a wide variety of methods for conducting such assessments, with little consensus in the field. The…

Concomitant with the rise of rationalizing accountability in higher education has been an increase in theoretical reflection about the forms accountability has taken and the ones it should take. The literature is now peppered by a wide array of distinctions (e.g. internal/external, inward/outward, vertical/horizontal, upward/downward,…

In this chapter, the author proposes a model for assessing institutional effectiveness. The Volkwein model for assessing institutional effectiveness consists of five parts that summarize the steps for assessing institutions, programs, faculty, and students. The first step in the model distinguishes the dual purposes of institutional effectiveness:…

Examination of 66 congruence studies, including benchmarking studies with improved methodology, showed that congruence is a sufficient though not a necessary condition for job satisfaction. A paradigmatic shift is needed with diverse research designs and methods, emphasizing experiment and drawing on person-environment psychology. (Contains 142…