This paper discusses the theoretical scope and practical applicability of generalizability (G) theory through the principle of symmetry. Major ideas are summarized and factors hindering applications of G theory in research conducted in French-speaking Europe are presented. The principle of symmetry affirms that any factor of a design can be…

The purpose of this study was to compare the scoring of Teacher Performance Assessment Instruments (TPAI) indicators using discrete descriptors when some are considered "essential" with the scoring of these same indicators, and when no descriptors are considered essential. The two questions addressed in this study were: (1) To what…

The assessment framework underlying a science performance monitoring program is process-oriented and intended to appeal to generalizability theory for a suitable estimation paradigm. Preliminary applications are described. Results suggest that computerized question-banking, domain-sampling of questions, and generalizablity theory together provide…

School test performance is commonly summarized in terms of the percentage of students at or above a cut score (PAAC) that has been set on a test. Two approaches to estimating the standard errors for school PAACs were examined in this study: conditional standard errors and overall standard errors. The tests used were English language arts and…

Results from a performance assessment in which 68 high school students wrote essays support the use of latent variable modeling for estimating reliability, concurrent validity, and generalizability of a scoring rubric. The latent variable modeling approach overcomes the limitations of certain conventional statistical techniques in handling…

Generalizability theory is used to examine the psychometric characteristics of the Listening and Writing Tests developed by American College Testing for its Work Keys program. Results with samples of 50 suggest the desirability of a minimum number of the tests' tape-recorded messages and the use of at least 2 raters. (SLD)

A meta-analysis was conducted of 107 classroom and laboratory studies concerning the effects of expecting a recall, recognition, essay, multiple-choice, or true-false test on students' subsequent achievement. Laboratory studies did not generalize well to classrooms. In classroom studies, subjects performed better when they knew which test type to…

There is a natural tension between experimentally designed studies and case studies, which differ in that they are not concerned with the interaction of variables in the quantitative and statistical sense. This paper describes a study that was not experimentally designed, but its major findings were generalizable to the overall population of…

Selected examples using the statistical packages Statistical Package for the Social Sciences (SPSS), the Statistical Analysis System (SAS), and BMDP are presented to facilitate their use and encourage appropriate uses in: (1) a hierarchical design; (2) a confounded factorial design; and (3) variance component estimation procedures. To illustrate…

An educational program on family resource management was evaluated to judge the goodness of fit of multiple regression equations for predicting program results across multiple sites. The applicability and effectiveness of the 11-lesson MONEY SENSE program was evaluated in six California locations for 190 clients. Indicators of effectiveness were:…

This study examined the difficulty level of essay topics used in the large-scale assessment of writing in relation to five different scoring models, and sought to determine what effects the scoring models would have on passing rates. In model one, examinee's score is the direct result of a score assigned by the reader or the sum of scores assigned…

Early childhood special educators recognize the necessity of establishing indices of reliability and validity for instruments that provide an index of developmental status. Many such instruments present little empirical evidence regarding psychometric integrity, particularly for a non-normative sample. The 341-item Battelle Developmental Inventory…

The consistency of ratings between two sections of a course taught by an instructor within a given semester was studied by examining sources of variation in rating instructor/course obtained through a college teacher evaluation system. Data were analyzed for five business school instructors at a Sogang University in Seoul (South Korea). Each…

This paper presents the results of ongoing analyses of the reliability of the System for Teaching and Learning Assessment and Review (STAR) as a comprehensive measure of classroom teaching and learning for making teacher certification decisions. Focus was on the effects of high stakes assessment conditions for certification on the generalizability…

Pronunciation is regarded as a valuable subskill in foreign language teaching and testing. Its quality is commonly assessed in a global way by having examinees read aloud. An atomistic test is a more systematic and explicit approach. Such a test would consist of about 40 items, use recorded performances, and draw on an inventory of pronunciation…