In this study, two approaches are used to assess the stability of Myers-Briggs Type Indicator scores across 3 administrations (N = 231): longitudinal configural frequency analysis with categorical scores and generalizability theory with the Preference Clarity Indices and continuous scores. The results are generally positive. Evaluation of…

It is incorrect to say "the test is reliable" because reliability is a function not only of the test itself, but of many factors. The present paper explains how different factors affect classical reliability estimates such as test-retest, interrater, internal consistency, and equivalent forms coefficients. Furthermore, the limits of classical test…

Unstable, and potentially invalid, variance component estimates may result from using only a limited portion of available data from operational performance assessments. However, missing observations are common in these settings because of the nature of the assessment design. This paper describes a procedure for overcoming the computational and…

The zeitgeist for reform in education precipitated a number of changes in assessment. Among these are performance assessments, sometimes linked to "high stakes" accountability decisions. In some instances, the trustworthiness of these decisions is based on variance components and error variances derived through generalizability theory.…

In comparing measurement theories, it is evident that the awareness of the concept of measurement error during the time of Galileo has lead to the formulation of observed scores comprising a true score and error (classical theory), universe score and various random error components (generalizability theory), or individual latent ability and error…

The present paper suggests that multivariate techniques are very important in social science research, and that canonical correlation analysis may be particularly useful. The logic of canonical analysis is explained and discussed. The necessity of using replicability/generalizability analyses is argued. It is suggested that cross-validation…

The use of Procrustean rotation as a procedure for assessing the invariance of study results is proposed. Researchers have long relied on significance testing as a measure of judging the worthiness of empirical findings. However, significance testing has come under fire because it does not provide information about the importance or replicability…

Invariance procedures have been known for over three decades. Through the years, these procedures have not been widely used by researchers. One reason for the lack of use is that most of the articles on invariance procedures have been mathematically oriented. This mathematical orientation and the lack of mathematical background on the part of most…

In the process of developing a conditionally-dependent item response theory (IRT) model, the problem arose of modeling an underlying multivariate normal (MVN) response process with general correlation among the items. Without the assumption of conditional independence, for which the underlying MVN cdf takes on comparatively simple forms and can be…

This paper describes how work by the United States Office of Personnel Management on the generalizability of employment test validities led to the development of a widely applicable meta-analysis method. The method focuses strongly on estimating the true variance of study correlations and effect size. This validity generalization procedure has…

The use of item response theory models for analyzing the results of judgmental standard setting studies (the Angoff technique) for establishing minimum pass levels is discussed. A comparison of three methods indicates the traditional approach may not be best. A procedure based on generalizability theory is suggested. (GDC)

Applied Schmidt-Hunter interactive validity generalization procedure to validity data for cognitive abilities tests for law enforcement occupations. Results indicated apparently lower validities and lesser generalizability for job criteria possibly due to low validity of the criterion and potential role of noncognitive factors. (Author/ABB)

Three essay questions were administered to junior education major college students. Factor analysis of the ratings showed that content played a large role in the students' responses, yielding dominant first order factors. Generalizability theory, used to examine the reliability of students' ratings, showed the need for more raters and questions.…

This study investigated the validity of the current reliability estimation procedure for the Test of Spoken English (TSE), a tape-mediated semi-performance test of 12 speaking tasks, from the perspective of generalizability theory and examined the feasibility of shortening the test without compromising the psychometric quality of the test. Data…

This study examined the effects of different test administration conditions on reading comprehension test scores. Evidence of performance differences across district testing conditions might imply that the meanings and interpretations associated with the corresponding test scores have limited generalizability (i.e., knowing how well one reads…