Rater-mediated activities in educational research occur when an expert judge or rater utilizes an instrument to judge persons or items and generates scale scores. Scale scores are from a subjective judgment and must undergo a quality control measure called rating quality. Rating quality in this study is broadly defined as the extent to which…

It is a much-lamented fact that research with the potential to inform or influence education policy instead remains policy inert. There are many reasons for this frustrating state of affairs, including a lack of strategic thinking on the part of researchers on how to successfully accomplish outreach--as opposed to communication with peers…

It has been argued that item response theory trait estimates should be used in analyses rather than number right (NR) or summated scale (SS) scores. Thissen and Orlando postulated that IRT scaling tends to produce trait estimates that are linearly related to the underlying trait being measured. Therefore, IRT trait estimates can be more useful…

Several current international comparative large-scale assessments of educational achievement (ICLSA) make use of "Rasch models", to address functions essential for valid cross-cultural comparisons. From a historical perspective, ICLSA and Georg Rasch's "models for measurement" emerged at about the same time, half a century ago. However, the…

Teacher and principal surveys are among the most common data collection techniques employed in education research. Yet there is remarkably little research on survey methods in education, or about the most cost-effective way to raise response rates among teachers and principals. In an effort to explore various methods for increasing survey response…

This paper introduces five commonly used approaches to analyzing multiple-choice test data. They are classical test theory, factor analysis, cluster analysis, item response theory, and model analysis. Brief descriptions of the goals and algorithms of these approaches are provided, together with examples illustrating their applications in physics…

Standardizing aspects of assessments has long been recognized as a tactic to help make evaluations of examinees fair. It reduces variation in irrelevant aspects of testing procedures that could advantage some examinees and disadvantage others. However, recent attention to making assessment accessible to a more diverse population of students…

The authors (a) discuss the importance of alignment for facilitating proper assessment and instruction, (b) describe the three most common methods for evaluating the alignment between state content standards and assessments, (c) discuss the relative strengths and limitations of these methods, and (d) discuss examples of applications of each…

Value-added models in education research allow researchers to explore how a wide variety of policies and measured school inputs affect the academic performance of students. Researchers typically quantify the impacts of such interventions in terms of "effect sizes", i.e., the estimated effect of a one standard deviation change in the…

This article examines how psychometric models based on certain assumptions have come to be used counterproductively by many practitioners in ways that limit the kinds of conclusions that can be made. The general problem of the context's influence on performance is discussed, and some implications are drawn. (SLD)

Educational and psychological measurement has been a main area of work for the Australian Council for Educational (ACER) since its inception. The theoretical and practical contributions of latent trait measurement and commentary on the relatively recent use of these models in Australia were the focus of a seminar celebrating the 50th anniversary…

The goal of test design is to devise an instrument that will provide a stable and accurate assessment of student ability in some area. One means of reaching this goal is through the use of latent trait models, which determine the relationship between the unobservable trait or ability and the observable test performance. Three common latent trait…

A study examined the results of an administration of a series of theoretically based prototype tests to 857 high school students in California, New York, and Wisconsin. By revising the existing framework of a prior study, tests were devised which attempted to measure three interrelated aspects of school literature: background knowledge, the…

Different models of performance assessment arise from interactions of three dimensions of assessment: the measurement versus the standards model of testing, quantitative and qualitative assumptions concerning the nature of learning, and whether learning and testing are situated or decontextualized. Addresses difficulties in implementing…

Classical procedures for calculating the two indices of decision consistency (P and Kappa) for criterion-referenced tests require two testings on each child. Huynh, Peng, and Subkoviak have presented one-testing procedures for these indices. These indices can be estimated without any test administration using Ebel's estimates of the mean, standard…