Generalizability theory is used to examine the sources of variability present in a teacher and course evaluation instrument. Two studies were conducted. In the first study, four different forms commonly used by one specific college of a large midwestern university were examined using responses of 915 students. The analysis of variance performed on…

Generalizability theory (g-theory) was used, as an alternative to classical test theory, to evaluate measurement error in a behaviorally anchored role-play measure, highlighting the usefulness of this theory in instrument development. G-theory partitions an observed score into the universe score and error scores associated with separate sources of…

Increasing social awareness of the need for accurate and fair testing has combined with technological developments to produce dramatic advances in psychometric models, methods, and procedures. Three of the most important new directions within the measurement field concern the following areas: (1) item response theory; (2) item bias or differential…

Three strategies for augmenting the interpretation of significance test results are illustrated. Determining the most suitable indices to use in evaluating empirical results is a matter of considerable debate among researchers. Researchers increasingly recognize that significance tests are very limited in their potential to inform the…

This study used generalizability theory and other statistical concepts to assess the application of the Angoff method to setting cutoff scores on two professional certification tests. A panel of ten judges gave pre- and post-feedback Angoff probability ratings of items of two forms of a professional certification test, and another panel of nine…

This document describes the development and assessment of a methodology for generating on-the-job-training (OJT) task proficiency assessment instruments. The Task Evaluation Form (TEF) development procedures were derived to address previously identified deficiencies in the evaluation of OJT task proficiency. The TEF development procedures allow…

Methods for examining the viability of assumptions underlying generalized linear models are considered. By appealing to the likelihood, a natural generalization of the raw residual plot for normal theory models is derived and is applied to investigating potential misspecification of the linear predictor. A smooth version of the plot is also…

Generalizability theory, which subsumes classical measurement theory as a special case, provides a general model for estimating the reliability of observational rating data by estimating the variance components of the measurement design. Research data from the "Heart Smart" health intervention program were analyzed as a heuristic tool.…

Examined the extent to which tutor ratings remained stable in the long term by evaluating 291 ratings of 140 tutors at Maastricht University in the Netherlands between 1992 and 1995. The results indicated that, if the aggregated score and overall judgement are used to interpret the precision of individual scores, four and two occasions,…

Objective criteria for evaluating the Eysenck Personality Inventory's internal structure are discussed. An approach based on targeted rotations and the test's scoring key is proposed as a means of providing common criteria. Data from earlier structure and test results for 195 undergraduates support the utility of 3 criteria developed. (SLD)

Oral retellings are strongly recommended as a way to measure reading comprehension for second language learners (Bernhardt, 1985, 1990, 1991). However, the reliability of such ratings is a matter of concern for a variety of reasons (Aiken, 1996; Cooper, 1981; Saal, Downey, & Lahey, 1980). The purpose of this study was to establish reliable rating…

The ECFMG[R] Clinical Skills Assessment (CSA[R]) was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise, conducted after a 15-minute interview with a standardized patient (SP), is specifically used to assess a candidate's…

Classical test theory, item response theory, and generalizability theory all treat the abilities to be measured as continuous variables, and the items of a test as independent probes of underlying continua. These models are well-suited to measuring the broad, diffuse traits of traditional differential psychology, but not for measuring the outcomes…

Tables of specifications are used to guide test developers in sampling items and maintaining consistency from form to form. This paper is a generalizability study of the American College Testing Program (ACT) Achievement Program Mathematics Test (AAP), with the content areas of the table of specifications representing multiple dependent variables.…

The Association of American Medical Colleges is conducting research to develop, implement, and evaluate a Medical College Admission Test (MCAT) essay testing program. Essay administration in the spring and fall of 1985 and 1986 suggested that additional research was needed on the development of topics which elicit similar skills and meet standard…