A classic topic in the fields of psychometrics and measurement has been the impact of the number of scale categories on test score reliability. This study builds on previous research by further articulating the relationship between item response theory (IRT) and classical test theory (CTT). Equations are presented for comparing the reliability and…

The problem of factor score indeterminacy implies that the factor and the error scores cannot be completely disentangled in the factor model. It is therefore proposed to compute Harman's factor score predictor that contains an additive combination of factor and error variance. This additive combination is discussed in the framework of classical…

This article introduces two new classification consistency indices that can be used when item response theory (IRT) models have been applied. The new indices are shown to be related to Rudner's classification accuracy index and Guo's classification accuracy index. The Rudner- and Guo-based classification accuracy and consistency indices are…

The majority of large-scale assessments develop various score scales that are either linear or nonlinear transformations of raw scores for better interpretations and uses of assessment results. The current formula for coefficient alpha (a; the commonly used reliability coefficient) only provides internal consistency reliability estimates of raw…

Cognitive diagnostic models (CDMs) model the probability of correctly answering an item as a function of an examinee's attribute mastery pattern. Because estimation of the mastery pattern involves more than a continuous measure of ability, reliability concepts introduced by classical test theory and item response theory do not apply. The cognitive…

An examinee-level (or conditional) reliability is proposed for use in both classical test theory (CTT) and item response theory (IRT). The well-known group-level reliability is shown to be the average of conditional reliabilities of examinees in a group or a population. This relationship is similar to the known relationship between the square of…

Large-scale testing programs involving classification decisions typically have multiple forms available and conduct equating to ensure cut-score comparability across forms. A test developer might be interested in the extent to which an examinee who happens to take a particular form would have a consistent classification decision if he or she had…

Evaluated coefficient alpha under violations of two classical test theory assumptions: essential tau-equivalence and uncorrelated errors through simulation. Discusses the interactive effects of both violations with true and error scores. Provides empirical evidence of the derivation of M. Novick and C. Lewis (1993). (SLD)

The clarification provided by Williams and Zimmerman on the reliability of gain scores is translated into recognizable patterns of change that tend to produce reliable or unreliable gain scores. The relevance of the traditional idea of reliability to the measurement of change is also discussed. (SLD)

Presents minimization and maximization models for parallel test construction under constraints. The minimization model constructs weakly and strongly parallel tests of minimum length, while the maximization model constructs weakly and strongly parallel tests with maximum test reliability. (Author/SLD)

Modified equations for the validity and reliability of difference scores that describe applied testing situations are examined. This examination reveals that simple gain scores can be more useful in research than has commonly been believed. Simple gain scores are neither inherently unreliable nor lack predictive validity. (SLD)

Reviews relevant aspects of generalizability theory related to performance assessments and discusses the role of various facets in assessing the generalizability of performance assessments. Also considers some popular estimates of reliability for performance assessments from the perspective of generalizability theory. (SLD)

Obtained asymptotic standard errors of item, test, and score information function estimates, and used numerical illustrations to show that the response function estimates are rather stable in spite of the unstable parameter estimates. However, information function estimates are shown to be relatively unstable. (SLD)

Describes an item response theory-based structural equation model that allows the short-term stability and the magnitude of retest effects to be assessed for some types of personality traits. Provides an empirical application of the model and discusses the substantive implications of the results. (SLD)

Compared approaches to determining the precision of gain scores: (1) the asymptotic normal distribution of the maximum likelihood estimator of the person parameter; and (2) the exact conditional distribution of the gain score. Use of three data sets illustrates that these methods yield more relevant and more detailed information than traditional…