Randomized A/B tests allow causal estimation without confounding but are often under-powered. This paper uses a new dataset, including over 250 randomized comparisons conducted in an online learning platform, to illustrate a method combining data from A/B tests with log data from users who were not in the experiment. Inference remains exact and…

Participatory modeling (PM) is an engaged research methodology for creating analog or computer-based models of complex systems, such as socio-environmental systems. Used across a range of fields, PM centers stakeholder knowledge and participation to create more internally valid models that can inform policy and increase engagement and trust…

Facebook is widely used and researched. However, though the data generated by educational technology tools and social media platforms other than Facebook have been used for research purposes, very little research has used Facebook posts as a data source--with most studies relying on self-report studies. While it has historically been impractical…

Qualitative studies in the domain of physics education research have become more common in the last several decades. Methodologically, this has been marked by an expansion of the types of data collected in physics education research (PER): namely, in the use of individual and group interviews, problem-solving sessions, and classroom…

OpenStax Kinetic is an innovative research infrastructure that aims to transform education and learning research in the digital age. With its access to large sample sizes, authentic learning environments, experimental control, scalability, security and privacy protection, Kinetic provides an unparalleled opportunity for researchers to study the…

While conflict event data sets are increasingly used in contemporary conflict research, important concerns persist regarding the quality of the collected data. Such concerns are not necessarily new. Yet, because the methodological debate and evidence on potential errors remains scattered across different subdisciplines of social sciences, there is…

Although educational research and evaluation generally occur in multilevel settings, many analyses ignore cluster effects. Neglecting the nature of data from educational settings, especially in non-randomized experiments, can result in biased estimates with long-term consequences. Our manuscript improves the availability and understanding of…

Despite the rising popularity of big data, there is speculation that evaluators have been slow adopters of these new statistical approaches. Several possible reasons have been offered for why this is the case: ethical concerns, institutional capacity, and evaluator capacity and values. In this method note, we address one of these barriers and aim…

Purpose: The purpose of this dissertation is to investigate the feasibility of using tests of robustness in peer review. This study involved selecting three high-impact papers which featured open data and utilized bioinformatic analyses but provided no source code and refactoring these to allow external survey participants to swap tools,…

This paper presents an ethical framework designed to support the development of critical data literacy for research methods courses and data training programmes in higher education. The framework we present draws upon our reviews of literature, course syllabi and existing frameworks on data ethics. For this research we reviewed 250 research…

Changing criterion designs (CCD) are single-case experimental designs that entail a step-by-step approximation of the final level desired for a target behavior. Following a recent review on the desirable methodological features of CCDs, the current text focuses on an analytical challenge: the definition of an objective rule for assessing the…

This conceptual article considers data used in the humanities and the human sciences, which are fundamentally different from data in other disciplines, such as the sciences or medicine. Data in the humanities are, however, equally important to study and understand. Humanists and others studying human artifacts often face the dual challenge of both…

This work proposes relevant ingredients to highlight the factors with significant impact on efficiency of research structures in higher education. The ingredients in question include methods and their implementation taking into account the choice and operationalization of factors as well as options of the retained methods. The methods employed in…

This article presents a pedagogical strategy that uses Chex Mix™ as a visual and physical means of helping students learn data analysis. The manipulative is used to guide students to a deeper level of analysis by looking "within" and "between" data sources. Specific steps are shared about using Chex Mix™ snacks to teach the…

Since the early 2000s, the proliferation of cameras, whether in mobile phones or CCTV, led to a sharp increase in visual recordings of human behavior. This vast pool of data enables new approaches to analyzing situational dynamics. Application is both qualitative and quantitative and ranges widely in fields such as sociology, psychology,…