Educational reform priorities such as emphasis on quantitative modelling (QM) have positioned undergraduate biology instructors as designers of QM experiences to engage students in authentic science practices that support the development of data-driven and evidence-based reasoning. Yet, little is known about how biology instructors adapt to the…

Attrition in longitudinal studies is a major threat to the representativeness of the data and the generalizability of the findings. Typical approaches to address systematic nonresponse are either expensive and unsatisfactory (e.g., oversampling) or rely on the unrealistic assumption of data missing at random (e.g., multiple imputation). Thus,…

Over the past 50 years, scientists have made amazing discoveries about the origins of human language acquisition. Central to this field of study is the process by which infants' perceptual sensitivities gradually align with native language structure, known as "perceptual narrowing." Perceptual narrowing offers a theoretical account of…

Experimental studies of educational interventions are rarely based on representative samples of the target population. This simulation study tests two formal sampling strategies for selecting districts and schools from within strata when they may not agree to participate if selected: (1) balanced selection of the most typical district or school…

The study of gender identity is becoming increasingly important to the field of higher education as younger generations of students enter their undergraduate programs. "As the number of college students identifying as transgender," gender nonconforming, or another gender identity, increases, "so too does the need to understand their…

Mathematical reasoning, a fundamental aspect of human cognition, poses significant challenges for artificial intelligence (AI) systems. Despite recent advancements in natural language processing (NLP) and large language models (LLMs), AI's ability to replicate human-like reasoning, generalization, and efficiency remains an ongoing research…

Much of learning analytics research has focused on factors influencing model generalizability of predictive models for academic performance. The degree of model generalizability across courses may depend on aspects, such as the similarity of the course setup, course material, the student cohort, or the teacher. Which of these contextual factors…

This article extends multivariate generalizability theory (MGT) to tests with different random-effects designs for each level of a fixed facet. There are numerous situations in which the design of a test and the resulting data structure are not definable by a single design. One example is mixed-format tests that are composed of multiple-choice and…

This paper aims to unsettle assumptions of generalizability and representativeness in quantitative research using queer framings and positionalities. We argue that generalizability and representativeness are tools of supremacist dominance that reinforce harmful and essentialist categories of identities for the false purpose of statistical…

The costs of an objective structured clinical examination (OSCE) are of concern to health profession educators globally. As OSCEs are usually designed under generalizability theory (G-theory) framework, this article proposes a machine-learning-based approach to optimize the costs, while maintaining the minimum required generalizability…

A conceptual confusion of "theory" exists in South African Open Distance and e-Learning (ODeL) research in which the concept is used with borderline, related, contrary, invented, or illegitimate concepts such as a model, approach, construct, hypothesis, theoretical framework, or conceptual framework. As a result, some researchers choose…

In this study, we examined the scoring and generalizability assumptions of an explicit instruction (EI) special education teacher observation protocol using many-faceted Rasch measurement (MFRM). Video observations of classroom instruction from 48 special education teachers across four states were collected. External raters (n = 20) were trained…

Many high-stakes examinations in the United Kingdom (UK) use both constructed-response items and selected-response items. We need to evaluate the inter-rater reliability for constructed-response items that are scored by humans. While there are a variety of methods for evaluating rater consistency across ratings in the psychometric literature, we…

Research Findings: Given the variable nature of early childhood settings, practitioners and researchers need better guidance on what conditions influence observations conducted within early childhood settings (National Research Council, 2008). Using 230 observations from 23 three- and four-year-old children, we conducted a Generalizability study…

Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the…