We contend that generalizability (G) theory allows the design of psychometric approaches to testing English-language learners (ELLs) that are consistent with current thinking in linguistics. We used G theory to estimate the amount of measurement error due to code (language or dialect). Fourth- and fifth-grade ELLs, native speakers of…

In this article, the author echoes his co-author's and colleague's pleasure (Hill, this issue) at the thoughtfulness and far-ranging nature of the comments to their initial attempts at test validation for the mathematical knowledge for teaching (MKT) measures using the validity argument approach. Because of the large number of commentaries they…

Formulating-Hypotheses (F-H) items present a situation and ask the examinee to generate as many explanations for it as possible. This study examined the generalizability, validity, and examinee perceptions of a computer-delivered version of the task. Eight F-H questions were administered to 192 graduate students. Half of the items restricted…

Recently, there has been a significant expansion in the use of performance assessment in large scale testing programs. Although there has been significant support from curriculum and policy stakeholders, the technical feasibility of large scale performance assessments has remained a question. This report is intended to contribute to the debate by…

Data from performance measures that were part of two foreign language teacher certification examinations were used in a generalizability study of the quality of their performance ratings. A total of 775 examinees from the Spanish K-12 and 192 examinees from the French K-12 subject area tests of the Florida Teacher Certification Examinations were…

This paper argues that special procedures for constructing assessment tools containing performance assessment tasks are unnecessary and that current test methodology can easily be generalized to complex performance assessment tasks without destroying the desirable characteristics of those tasks. Reasonable statistical requirements for sound…

Most training programs in education and psychology focus on classical test theory techniques for assessing score dependability. This paper discusses generalizability theory and explores its concepts using a small heuristic data set. Generalizability theory subsumes and extends classical test score theory. It is able to estimate the magnitude of…

Classroom researchers are frequently urged to provide evidence of the reliability of their data. In the case of observational data, three approaches to this have emerged: observer agreement, generalizability theory, and measurement error. Generalizability theory provides the most powerful approach given an adequate data collection design, but…

The purpose of this study was to provide evidence for adapting and generalizing the use of the Children's Academic Motivation Inventory (CAMI) to high school students. The instrument was originally developed to provide a reliable, valid, theory-based measure of academic achievement motivation; it was suitable for use with children aged 12-14…

The application of generalizability theory to the reliability and error variance estimation for performance assessment scores is discussed. Decision makers concerned with performance assessment need to realize the restrictions that limit generalizability such as limitations that lead to reductions in the number of tasks possible, rater quality,…

Issues of dimensionality, validity, reliability, and generalizability of students' ratings of instruction were studied with 610 college students who evaluated their instructors on the Teacher Effectiveness Scale. The Scale had high internal consistency, but interrater reliability, generalizability, and predictive validity warrant further study.…

The framework and procedures of generalizability theory are introduced and illustrated in this instructional module that uses a hypothetical scenario involving writing proficiency. Generalizability analyses are useful for understanding the relative importance of various sources of error and for designing efficient measurement procedures. (SLD)

The monograph by D. Rogosa and G. Ghandour represents a body of cohesive and comprehensive research that can be the basis of a new measurement theory combining features of generalizability theory and strong true-score theory. Principles, approaches, arguments, and conclusions are reviewed; and critical comments are offered. (SLD)

Issues raised with the statistical models developed are discussed point by point, restating the emphasis on finite observation time, and reiterating the criticism of traditional psychometric methods. It is noted that the language and technical formulation of psychometrics can be extremely awkward in dealing with biased estimates. (SLD)

Investigates whether training raters to interpret the scoring dimensions on a rubric would increase reliability. Compares two groups of kindergarten and first-grade teachers: one group with training, one without. Finds that training increases raters' abilities to reliably interpret scoring items. (SC)