Using a sample of 70,399 published p-values from 192 meta-analyses, we empirically estimate the counterfactual distribution of p-values in the absence of any biases. Comparing observed p-values with counterfactually expected p-values allows us to estimate how many p-values are published as being statistically significant when they should have been…

Fetters et al.'s (2013) mixed methods integration framework uses construction metaphors: building, connecting, merging, and embedding. In a similar vein, this article uses an architectural metaphor and introduces design patterns as building blocks for mixed methods research design. A design pattern embeds one specific design decision into its…

This paper argues for three aspects of tolerance with respect to QCA research: tolerance with respect to different approaches to QCA; producing QCA research with tolerance (work that is resistant to criticism); and for QCA researchers to be clear about the tolerance of the solutions they present -- especially in terms of calibration and truth…

Researchers increasingly take advantage of the comparative case design to build theory, but the degree of case dependence is occasionally discussed and theorized. We suggest that the comparative case study design might be subject to an often underappreciated threat--dependence across cases--under certain conditions. Using research on innovation…

This study aims to introduce network meta-analysis (NMA) to provide educational researchers with an extended view of the reviewing educational research. Meta-analytic methods have been widely used in educational research reviews. However, weaknesses have emerged in the multi-group comparison analysis of educational studies where different…

Recent years have witnessed a host of innovations for conducting research with qualitative comparative analysis (QCA). Concurrently, important issues surrounding its uses have been highlighted. In this article, we seek to help users design QCA studies. We argue that establishing inference with QCA involves three intertwined design components:…

This article develops new closed-form variance expressions for power analyses for commonly used difference-in-differences (DID) and comparative interrupted time series (CITS) panel data estimators. The main contribution is to incorporate variation in treatment timing into the analysis. The power formulas also account for other key design features…

As a research field geared toward understanding and improving learning, Learning Analytics (LA) must be able to provide empirical support for causal claims. However, as a highly applied field, tightly controlled randomized experiments are not always feasible nor desirable. Instead, researchers often rely on observational data, based on which they…

Designing and interpreting controlled experiments are important inquiry skills addressed in many current science curricula. The relevant skills associated with the design and interpretation of controlled experiments are summarised under the term control-of-variables strategy (CVS). Research on elementary school students' CVS skills shows that they…

Inquiry-based laboratories were implemented into a General Chemistry Laboratory sequence, and the impact of these exercises on students' experimental design skills was assessed using a four-part assessment developed for this study. This assessment contained a multiple-choice section, a section asking students to explain their reasoning behind a…

The purpose of this comparative case study is to explore the recent use of phenomenological inquiries in music education articles published in five selected journals listed in Thomson Reuters Web of Science between 2012 and 2015. The five music education journals are "The British Journal of Music Education," "Bulletin of the Council…

Comparison studies conducted to determine which instructional interventions are most efficient for teaching discrete behaviors to individuals with disabilities are potentially valuable, although some threats to internal validity may be more likely in these studies. Studies included in this review typically met common internal validity standards,…

Background: Over the past 20 years, much has been written about the merits and limitations of matching methods, particularly propensity score matching (see, e.g., Stuart, 2010 for a review of matching approaches), and the myriad ways matching can be implemented to identify a comparison group that is similar to the treatment group on observable…

We conducted a systematic literature review of 111 empirical studies published between 1969 and 2018. We found that research does not exponentially grow in comparison to research on higher education or leadership; empirical research emerging in the 1980s does not completely replace non-empirical inquiry. Studies from English-speaking authors…

The Benefit Offset National Demonstration (BOND) tested a $1 reduction in benefits per $2 earnings increase above the level at which Social Security Disability Insurance benefits drop from full to zero under current law. BOND included a rare and large "population-representative" experiment: It applied the rule to a nationwide, random…