In cluster randomized evaluations, a treatment or intervention is randomly assigned to a set of clusters each with constituent individual units of observations (e.g., student units that attend schools, which are assigned to treatment). One consideration of these designs is how many units are needed per cluster to achieve adequate statistical…

Randomized A/B tests allow causal estimation without confounding but are often under-powered. This paper uses a new dataset, including over 250 randomized comparisons conducted in an online learning platform, to illustrate a method combining data from A/B tests with log data from users who were not in the experiment. Inference remains exact and…

Large-scale, classroom-based experiments using adaptive instructional software pose somewhat unique challenges for experimental design and deployment. One reason for this is that adaptive software allows students to advance through the curriculum at different rates and encounter content at different times, meaning that content targeted for…

Virtual experiments offer opportunities for students to engage in the scientific process and observe scientific phenomena. However, virtual experiments are not always effective for learning. The current study examined how two small manipulations might support better learning from virtual experiments and the extent to which this varied across…

Cognitive modelling shares many features with statistical modelling, making it seem trivial to borrow from the practices of robust Bayesian statistics to protect the practice of robust cognitive modelling. We take one aspect of statistical workflow--prior predictive checks--and explore how they might be applied to a cognitive modelling task. We…

As online learning platforms become more ubiquitous throughout various curricula, there is a growing need to evaluate the effectiveness of these platforms and the different methods used to structure online education and tutoring. Towards this endeavor, some platforms have performed randomized controlled experiments to compare different user…

For researchers seeking to improve education, a common goal is to identify teaching practices that have causal benefits in classroom settings. To test whether an instructional practice exerts a causal influence on an outcome measure, the most straightforward and compelling method is to conduct an experiment. While experimentation is common in…

Theories regarding the long-term effects of educational interventions are often assumed, but rarely tested using experimental methods. In the following commentary, we argue that the shortage of randomized control trials with long-term follow-up presents serious problems for the field, as it hampers our ability to develop educational programs that…

This paper investigates whether voice assistants can play a useful role in the specialized work-life of the knowledge worker (in a biology lab). It is motivated both by promising advances in voice-input technology, and a long-standing vision in the community to augment scientific processes with voice-based agents. Through a reflection on our…

To build knowledge, separate yet related learning episodes can be integrated with one another and then used to derive new knowledge. Separate episodes are often experienced through different formats, such as text passages and graphic representations. Accordingly, in the present research, we tested integration of learning episodes provided through…

In five experiments, we examined the conditions under which participants remembered true and false information given as feedback. Participants answered general information questions, expressed their confidence in the correctness of their answers, and were given true or false feedback. In all five experiments, participants hyper'corrected' when…

Statistical mediation allows researchers to investigate potential causal effects of experimental manipulations through intervening variables. It is a powerful tool for assessing the presence and strength of postulated causal mechanisms. Although mediation is used in certain areas of psychology, it is rarely applied in cognitive psychology and…

Randomized controlled trials (RCTs) admit unconfounded design-based inference--randomization largely justifies the assumptions underlying statistical effect estimates--but often have limited sample sizes. However, researchers may have access to big observational data on covariates and outcomes from RCT non-participants. For example, data from A/B…

Science assessments should evaluate the full complement of inquiry practices (NGSS, 2013). Our previous work has shown that a large proportion of students' open responses did not match their scientific investigations (Li et al., 2017a). The present study both unpacks and compares the sub-components underlying students' performance for…

Early childhood science and engineering education offer a prime context to foster approaches-to-learning (ATL) and executive functioning (EF) by eliciting children's natural curiosity about the world, providing a unique opportunity to engage children in hands-on learning experiences that promote critical thinking, problem solving, collaboration,…