Used the domain sampling model of generalizability theory to study ways of improving a construct measure, the Achievement Motives Scale (AMS) (R. Nygard and T. Gjesme, 1973). AMS results for 504 Chinese high school students and 252 U.S. graduate students are the basis for suggestions to improve the two language versions' approximations to the…

Generalizability of performance-based assessment scores across raters and tasks is examined, focusing on implications of generalizability analyses for specific uses and interpretations of assessment results. Although it seems probable that assessment conditions, task characteristics, and interactions with instructional experiences affect the…

Examined the generalizability and validity and examinee perceptions of a computer-delivered version of 8 formulating-hypotheses tasks administered to 192 graduate students. Results support previous research that has suggested that formulating-hypotheses items can broaden the abilities measured by graduate admissions measures. (SLD)

The internal/external frame of reference (I/E) model explains a seemingly paradoxical pattern of relations between math and verbal self-concepts and corresponding measures of achievement, extends social comparison theory, and has important educational implications. In a cross-cultural study of nationally representative samples of 15-year-olds from…

This article presents a standard-setting procedure for performance assessment in a foreign language, through which some of the major problems facing performance assessment in criterion-referenced testing can be addressed. The procedure, which was geared to revealing and accommodating inter-judge variability, employed the synergy of multiple…

There is increasing pressure, both from institutions central to the national scientific mission and from regional and national accrediting agencies, on natural sciences faculty to move beyond course examinations as measures of student performance and to instead develop and use reliable and valid authentic assessment measures for both individual…

This study describes the use of generalizability theory (GT) and many-facet Rasch measurement (MFRM) to evaluate psychometric properties of responses obtained from an assessment designed to measure complex problem-solving skills. The assessment revolved around the school activity of kickball. The task required of each student was to decide on a…

Purpose: Determining the valid and fair use of the interview for medical school admissions is contingent upon a demonstration of the reproducibility of interview scores. This study seeks to establish the generalizability of interview scores, first assessing the existing research evidence, and then analyzing data from a non-experimental independent…

Three studies were conducted using the interpersonal grid, a method for assessing perceptions of agentic and communal behavior based on the interpersonal circumplex. The 1st examined consistency across perceivers and convergence between perceiver and the perceived person. The 2nd examined whether responses to the interpersonal grid were sensitive…

This study demonstrates the advantages of using a constrained optimization algorithm to explore the optimal number of prompts, modes of discourse, and raters for achieving an acceptable level of reliability during a direct writing assessment. Writing samples elicited from 50 college students were rated by 3 graduate students and the scores…

The application of mathematical programming techniques is extended to the construction of measurement instruments in generalizability theory. Key concepts in generalizability theory are explained and a description is given of: (1) the one-facet crossed design; (2) the two-facet crossed design; and (3) a two-facet nested design. The optimization of…

Although student evaluation data provide a reasonable basis for making decisions about instructors when generalizing across courses and students, when the course is the object of measurement (OM), data are less generalizable. This finding may be due to the type of evaluation items used or to academic discipline differences in the type of courses…

As schools move toward performance assessment, there is increasing discussion of using these assessments for accountability purposes. When used for making decisions, performance assessments must meet high standards of validity and reliability. One major source of unreliability in performance assessments is interrater disagreement. In this paper,…

The purpose of this study was to investigate the impact of local item dependence (LID) in passage-based testlets on the test score reliability of an English as a Foreign Language (EFL) reading comprehension test from the perspective of generalizability (G) theory. Definitions and causes of LID in passage-based testlets are reviewed within the…

This study examined the generalizability and dependability of a performance-based assessment in algebra. Four forms of a five-item test were constructed using different subsets of eight items based on attributes from task analysis. Subjects included 142 "algebra 2" students from 2 high schools in the Midwestern United States and 148 11th…