The U.S. Department of Education (Department) is committed to supporting the use of technology to improve teaching and learning and to support innovation throughout educational systems. This report addresses the clear need for sharing knowledge and developing policies for "Artificial Intelligence," a rapidly advancing class of…

Assessments of natural language can provide vast information about individuals' thoughts and cognitive process, but they often rely on time-intensive human scoring, deterring researchers from collecting these sources of data. Natural language processing (NLP) gives researchers the opportunity to implement automated textual analyses across a…

This introductory article describes how constructed response scoring is carried out, particularly the rater monitoring processes and illustrates three potential designs for conducting rater monitoring in an operational scoring project. The introduction also presents a framework for interpreting research conducted by those who study the constructed…

The same L2 speaking performance may be analysed and evaluated in very different ways by different teachers or raters. We present a new, technology-assisted research design which opens up to investigation the trajectories of convergence and divergence between raters. We tracked and recorded what different raters noticed when, whilst grading a…

In this special issue on high-stakes English language testing in China, the two articles on computer-based testing (Jin & Yan; He & Min) highlight a number of consistent, ongoing challenges and concerns in the development and implementation of the nationwide IB-CET (Internet Based College English Test) and institutional computer-adaptive…

This new ACT publication is an annual report offering meaningful research insights for some of the most pressing questions impacting admissions and enrollment practice. In the first release of this report, ACT research sheds light on the following topics: (1) the practice of super-scoring; (2) STEM major choice; (3) factors impacting retention and…

This report summarizes the complex research literature on quantitative methods for assessing how impacts of educational interventions on instructional practices and student learning differ across students, educators, and schools. It also provides technical guidance about the use and interpretation of these methods. The research topics addressed…

This document describes the breadth of the research that the ETS (Educational Testing Service) Research & Development division is conducting in 2010. This portfolio will be updated in early 2011 to reflect changes to existing projects and new projects that were added after this document was completed. The research described in this portfolio falls…

States and districts can use this tool to create their own plans for continual improvement of an observation system, no matter where they are in their implementation. Included are action steps to improve observation rubrics, observer training, observer assessment, and monitoring. A planning process is described to assess current status, determine…

This document describes the breadth of the research being conducted in 2008 by the Research and Development Division at Educational Testing Service (ETS). The research described falls into three large categories: (1) Research supported by the ETS research allocation; (2) Research funded by testing programs at ETS; and (3) Research funded by…

The "PISA 2012 Technical Report" describes the methodology underlying the PISA 2012 survey, which tested 15-year-olds' competencies in mathematics, reading and science and, in some countries, problem solving and financial literacy. It examines the design and implementation of the project at a level of detail that allows researchers to…

The National Assessment of Educational Progress (NAEP) and its reports are a key measure in informing the nation on how well the goal of scientific literacy for all students is being met. The "Science Framework for the 2011 National Assessment of Educational Progress" sets forth the design of the NAEP Science Assessment. The 2011 NAEP…

Recent articles in "The Journal for Specialists in Group Work" have discussed credibility indicators for quantitative and qualitative studies (Asner-Self, 2009; Rubel & Villalba, 2009). This article extends upon these contributions by discussing measurement issues that are relevant to producers and consumers of quantitative group research. This…

This report discusses the most relevant issues concerning student standardised testing in which there are no-stakes for students ("standardised testing") through a literature review and a review of the trends in standardised testing in OECD countries. Unlike standardised tests in which there are high-stakes for students, no-stakes implies that…

This article discusses the use of propensity scoring in experimental program evaluations to estimate impacts for subgroups defined by program features and participants' program experiences. The authors discuss estimation issues and provide specification tests. They also discuss the use of an overlooked data collection design--obtaining predictions…