The focus of the present study is on students' experiences of enjoyment, an emotion largely neglected in educational research. We present a model in which specific levels of generalization of the construct of enjoyment are differentiated. Based on their extent of generalization, these differentiated constructs of enjoyment are located in a…

Although multivariate generalizability theory was developed more than 30 years ago, little published research utilizing this framework exists and most of what does exist examines tests built from tables of specifications. In this context, it is assumed that the universe scores from levels of the fixed multivariate facet will be correlated, but the…

Generalizability theory (G-theory) and many-facet Rasch measurement (Rasch) manage the variability inherent when raters rate examinees on test items. The purpose of G-theory is to estimate test reliability in a raw score metric. Unadjusted examinee raw scores are reported as measures. A variance component is estimated for the examinee…

Possible integrated and independent tasks were pilot tested for the writing section of a new generation of TOEFL® (Test of English as a Foreign Language™) examination. This study examines the impact of various rating designs as well as the impact of the number of tasks and raters on the reliability of writing scores based on integrated and…

Rating scales are a typical method for evaluating a student's performance in outcomes assessment. The analysis of the quality of information from rating scales poses special measurement problems when researchers work with faculty in their development. Generalizability measurement theory offers a set of techniques for estimating errors or…

An evaluation system which would describe and assess statewide services for preschool children is described. Component theory conceptualizes the unit of analysis for evaluation as the component. This approach increases the generalizability and utilization of evaluations and enhances the ability to evaluate several programs at the state level.…

Compared with data from civilian institutions, data from two graduating classes at the United States Military Academy showed extrordinary stability of independently calculated grade point averages from freshman to senior years and no decline in the validity of Scholastic Aptitude Tests and high school class rank as predictors of these GPAs over…

There is a unity underlying the diversity of models for the analysis of multivariate data. Essentially, they constitute a family of models, most generally nonlinear, for structural/functional relations between variables drawn from a behavior domain. (Author)

The reliability of criterion-referenced tests (CRTs) used in health program evaluation can be conceptualized in different ways. Formulas are presented for estimating appropriate standard error of measurement (SEM) for CRTs. The SEM can be used in computing confidence intervals for domain score estimates and for a cut-score. (Author/LMO)

The evidence for the psychometric adequacy of behavioral interviews in terms of traditional psychometric theory and generalizability theory are reviewed. The review resulted in the conclusion that behavioral interviews have some evidence for interrater reliability, content validity, and criterion-related validity. Additional research in several…

In this study, the reliability of the American College Testing (ACT) Program's "Survey of Academic Advising" (SAA) was examined using both univariate and multivariate generalizability theory approaches. The primary purpose of the study was to compare the results of three generalizability theory models (a random univariate model, a mixed…

Writing fits well within the realm of outcomes suitable for observation by performance assessments. Studies of the reliability of performance assessments have suggested that interrater reliability can be consistently high. Scoring consistency, however, is only one aspect of quality in decisions based on assessment results. Another is…

Through the standard error, rather than a reliability coefficient, generalizability theory provides an indicator of the uncertainty attached to school and individual scores on performance assessments. Recommendations are made to apply generalizability theory to current performance assessments, emphasizing practices that differ from usual…

Description of common errors found in the statistical methodologies of research carried out by librarians, focuses on sampling and generalizability. The discussion covers the need to either adapt library research to the statistical abilities of librarians or to educate librarians in the proper use of statistics. (15 references) (CLB)

A methodology is presented for minimizing mean error variance in generalizability studies when resource constraints are imposed. The optimal number of observations and conditions of facets for random model, fully crossed one- and two-facet designs can be decided. Parallel closed form formulas can be determined for other designs. (SLD)