A measurement approach called generalizability theory (G-theory) is an important alternative to the more familiar classical measurement theory that yields less useful coefficients such as alpha or the KR-20 coefficient. G-theory is a theory about the dependability of behavioral measurements that allows the simultaneous estimation of multiple…

Psychometric theory has been one of psychology's stronger foundations and a major contributor to the recognition of psychology as a scientific discipline. Basic principles of psychometric theory led to the development of respected intelligence tests and large and comprehensive testing and assessment programs. This paper synthesizes major…

The Stages of Concerns Questionnaire (SoCQ) is an established instrument which has been used primarily with inservice teachers involved in innovations. While it focuses on inservice teachers' concerns about an innovation, the instrument is based on theoretical constructs developed by the work of Frances Fuller and others during the 1960s with…

The purpose for which a test is used and the examinees' stage of learning are two anchor points that are incorporated into a suggested framework for conducting construct validation studies for tests of students with English as a second language (ESL) or English as a foreign language (EFL). The framework includes the use of generalizability theory,…

The central thesis of this article is that the process of eliminating validity threats depends fundamentally on no-difference findings, a fact that has not been made explicit by researchers. The implications of this neglect are explored using examples from a number of different substantive areas such as psychology, health, and medicine.…

This study examined whether two variations on the typical Angoff group standard-setting process would produce sufficiently consistent results to recommend their use. The results imply that judgments gathered after an initial traditional group-process session can provide an efficient alternative mechanism for setting cutting scores using the Angoff…

This study examined the impact of rater agreement on decisions concerning the alignment between the Golden State Examination (GSE) in High School Mathematics and the University of California (UC) "Statement on Competencies in Mathematics." UC faculty and high school mathematics teachers (n = 20) rated the mathematics items of the GSE…

The peer review system in professional publications of scientific information is discussed in four articles. Topics discussed include the effects of information technology on scholarly communication channels, the publish or perish syndrome, the incidence of misrepresentation in scientific research, reasons for increasing difficulties in…

It is proposed that criteria for evaluating the quality of performance should be defined, at least tentatively, during the initial design of a performance assessment. Six characteristics of sound criteria are (1) significance; (2) fidelity; (3) generalizability; (4) developmental appropriateness; (5) accessibility; and (6) utility. (SLD)

Evidence is presented on the generalizability and convergent validity of performance assessments using data from six studies of student achievement that sampled a wide range of measurement facets and methods. Results at individual and school levels indicate that task-sampling variability is the major source of measurement error. (SLD)

Examined the generalizability of written-response scores on the English 30 diploma examination administered to Alberta 12th-grade students. Student scores differed as a function of rater, but this variance component was small across two tasks and two administrations; score generalizability was high using a two-rater system; and scale variability…

Used generalizability theory to assess the variation in school scores across very short test forms that measured mathematics scores in grades 4 and 8. More than 25,000 students took each form of the 3 tests for each grade. Results demonstrate the lack of comparability in school scores across short, nonparallel tests forms and the importance of…

Generalizability (G) theory was used with a sample of 37 third-grade students to assess the variability in words correct per minute (WCPM) scores caused by student skill and passage variability. Reliability-like coefficients and the SEM based on a specific number of assessments using different combinations of passages demonstrated how manipulating…

The authors present results of a generalizability study of running record assessment. They conducted 2 decision studies to ascertain the number of raters and passages necessary to obtain a reliable estimate of a student's reading ability on the basis of a running record assessment. Ten teachers completed running record assessments of 10…

Increased demands for colleges and universities to engage in outcomes assessment for accountability purposes have accelerated the need to bridge the gap between higher education practice and the fields of measurement, assessment, and evaluation. The "Handbook on Measurement, Assessment, and Evaluation in Higher Education" provides higher…