This paper lays out a framework for comparing the qualities and the quantities of information about student competence provided by multiple-choice and free-response test items. After discussing the origins of multiple-choice testing and recent influences for change, the paper outlines an "inference network" approach to test theory, in…

The basic equations of item response theory provide a foundation for inferring examinees' abilities and items' operating characteristics from observed responses. In practice, though, examinees will usually not have provided a response to every available item--for reasons that may or may not have been intended by the test administrator, and that…

A closed form approximation is given for the variance of examinee proficiency estimates in the Rasch model for dichotomous items, under the condition that only estimates, rather than true values, of item difficulty parameters are available. The term that must be added to the usual response-sampling variance is inversely proportional to both the…

Simultaneous estimation of many parameters can often be improved, sometimes dramatically so, if it is reasonable to consider one or more subsets of parameters as exchangeable members of corresponding populations. While each observation may provide limited information about the parameters it is modeled directly in terms of, it also contributes…

This paper synthesizes ideas from the fields of graphical modeling and education testing, particularly item response theory (IRT) applied to computerized adaptive testing (CAT). Graphical modeling can offer IRT a language for describing multifaceted skills and knowledge, and disentangling evidence from complex performances. IRT-CAT can offer…

This paper reviews recent work in factor analysis of categorical variables. Emphasis is on the generalized least squares solution. A section on maximum likelihood solution focuses on extensions of the classical model, especially the normal case. Many of the recent developments have taken place within this context, and it provides a unified…

Multiple matrix sampling (MMS) theory indicates how data may be gathered to most efficiently convey information about levels of attainment in a population, but standard analyses of these data require random sampling of items from a fixed pool of items. This assumption proscribes the retirement of flawed or obsolete items from the pool as well as…

Assuming vectors of item responses depend on ability through a fully specified item response model, this paper presents maximum likelihood equations for estimating the population parameters without estimating an ability parameter for each subject. Asymptotic standard errors, tests of fit, computing approximations, and details of four special cases…

Educational assessment concerns inference about student knowledge, skills, and accomplishments. Test theory has evolved in part to address questions of weight, coverage, and import of data. Resulting concepts and techniques can be viewed as applications of more general principles for inference in the presence of uncertainty. (SLD)

The precision of item parameter estimates can be increased by taking advantage of dependencies between the latent proficiency variable and auxiliary examinee variables such as age, courses taken, and years of schooling. Score gains roughly equivalent to two to six additional item responses can be expected in typical educational and psychological…

Recent developments in cognitive and educational psychology, such as increased appreciation of the situated nature of learning and understanding, call for broader ranges of student models and types of data than those standard in testing today. We must specify how what we observe on the test is related to competence as we conceptualize it, and…

Standard procedures for estimating item parameters in Item Response Theory models make no use of auxiliary information about test items, such as their format or content, or the skills they require for solution. This paper describes a framework for exploiting this information, thereby enhancing the precision and stability of item parameter…

California Assessment Program's application of matrix sampling and item response curve theory to the scaling and reporting of state assessment data is described. It is designed to express educational outcomes in an efficient and interpretable form that is both immediately informative and suited to analysis over extended periods of time.…

Scaling methodologies used in analysis of National Assessment of Educational Progress (NAEP) surveys since 1984 are reviewed. The plausible values methodology developed for NAEP scale-score analyses is described in the contexts of item response theory and average response method scaling. Current NAEP research in scaling is discussed. (SLD)

This paper illustrates how, in the item-response theory framework, collateral information about test items can augment or replace examinee responses when linking or equating new tests to established scales, using data from the Pre-Professional Skills Test for approximately 40,000 examinees. Collateral information can predict item operating…