The Research Excellence Framework is a high-stakes exercise used by the UK government to allocate billions of pounds of quality-related research (QR) funding and used by the media to rank universities and their departments in national league tables. The 2008, 2014 and 2021 assessments were zero-sum games in terms of league table position because…

The U.S. Department of Education (Department) is committed to supporting the use of technology to improve teaching and learning and to support innovation throughout educational systems. This report addresses the clear need for sharing knowledge and developing policies for "Artificial Intelligence," a rapidly advancing class of…

Due to myriad applications of the nominal group technique (NGT), a highly flexible iterative focus group method, researchers know little about its optimal scoring procedures. Exploring benefits and biases that such procedures might present, we aim to clarify how NGT scoring systems can privilege consensus or prioritization. In conducting the first…

As part of the Community College Research Center's (CCRC) Analysis of Statewide Developmental Education Reform Learning Assessment Study, Wake Technical Community College's partner team believes the recommendation and implications in the CCRC's Developmental Reading and English Assessment in a Researcher-Practitioner Partnership. Working Paper No.…

This article provides a validation framework for research on the development and use of science Learning Progressions (LPs). The framework describes how evidence from various sources can be used to establish an interpretive argument and a validity argument at five stages of LP research--development, scoring, generalisation, extrapolation, and use.…

This article provides a validation framework for research on the development and use of science Learning Progressions (LPs). The framework describes how evidence from various sources can be used to establish an interpretive argument and a validity argument at five stages of LP research--development, scoring, generalisation, extrapolation, and use.…

Randomized experiments are considered the gold standard for causal inference because they can provide unbiased estimates of treatment effects for the experimental participants. However, researchers and policymakers are often interested in using a specific experiment to inform decisions about other target populations. In education research,…

Although inter-rater reliability is an important aspect of using observational instruments, it has received little theoretical attention. In this article, we offer some guidance for practitioners and consumers of classroom observations so that they can make decisions about inter-rater reliability, both for study design and in the reporting of data…

In this special issue on high-stakes English language testing in China, the two articles on computer-based testing (Jin & Yan; He & Min) highlight a number of consistent, ongoing challenges and concerns in the development and implementation of the nationwide IB-CET (Internet Based College English Test) and institutional computer-adaptive…

Although International Association for the Evaluation of Educational Achievement-pioneered international large-scale assessment (ILSA) of education is now a well-established science, non-practitioners and many users often substantially misunderstand how large-scale assessments are conducted, what questions and challenges they are designed to…

The focus of the learning analytics community bridges the gap between controlled educational research and data mining. Online learning platforms can be used to conduct randomized controlled trials to assist in the development of interventions that increase learning gains; datasets from such research can act as a treasure trove for inquisitive data…

The goal of this research is to make sensitivity analysis accessible not only to empirical researchers but also to the various stakeholders for whom educational evaluations are conducted. To do this it derives anchors for the omitted variable (OV)-program participation association intrinsically, using the Love plot to present a wide range of…

This new ACT publication is an annual report offering meaningful research insights for some of the most pressing questions impacting admissions and enrollment practice. In the first release of this report, ACT research sheds light on the following topics: (1) the practice of super-scoring; (2) STEM major choice; (3) factors impacting retention and…

As an alternative to rubric scoring, comparative judgment generates essay scores by aggregating decisions about the relative quality of the essays. Comparative judgment eliminates certain scorer biases and potentially reduces training requirements, thereby allowing a large number of judges, including teachers, to participate in essay evaluation.…

This study aimed to gain knowledge of students' beliefs and difficulties in understanding p-values, and to use this knowledge to develop improved teaching programs. This study took place over four consecutive teaching semesters of a one-semester tertiary statistics unit. The study was cyclical, in that the results of each semester were used to…