Results of a study of the reliability of student ratings of academic advising from 15 postsecondary institutions show that generalizability theory is an appropriate and accurate way to estimate reliability. Advantages and disadvantages of the generalizability theory approach in comparison with coefficient alpha are discussed. (SLD)

This commentary focuses on the application of D. Rogosa and G. Ghandour's work to observational research on classroom processes. Rogosa and Ghandour have shown that the short length of an observation is typically the dominant source of error. Investigators should conduct observations for as long as possible. (SLD)

This study tests the generalizability of the factor pattern, structural parameters, and latent mean structure of a multidimensional, hierarchical model of physical self-concept in adolescents across gender and grade. A children's version of the Physical Self-Perception Profile (C-PSPP) was administered to seventh-, eighth- and ninth-grade high…

Although the rubric has emerged as one of the most popular assessment tools in progressive educational programs, there is an unfortunate dearth of information in the literature quantifying the actual effectiveness of the rubric as an assessment tool "in the hands of the students." This study focuses on the validity and reliability of the rubric as…

A total of 2,000 essays written by 1,000 students was submitted to generalizability analyses for domain-referenced tests. Each student had written one essay on each of two prompts representing two models of discourse. Each essay was read by six readers and judged on a scale of from 1 to 4. No reader read essays from both prompts. Reader agreement…

Two new indices of stability of content-referenced standard-setting results are presented, relating variability of judges' decisions to the variability of candidate scores and to the reliability of the test. These indices are used to indicate whether scores resulting from a standard-setting study are of sufficient precision. (Author/LMO)

An investigation of the Conners' Teacher Rating Scale-Revised hyperactivity scale found that the referents for teacher ratings should be determined, teachers' ratings should be made more objective, standardization across teachers should be demonstrated before norms are preferred, and the rating scale should be validated via observations or other…

In behavior research using cognitive and affective measures, there is often incongruity between the statistical analysis employed and the intended inference. This paper argues that incorporating items as levels of a random facet via generalizability theory allows the statistical examination of the inferential question in the desired universe of…

This study examines the score dependability of writing and speaking assessments from the Test of English as a Foreign Language (TOEFL) from the perspectives of univariate and multivariate generalizability theory (G-theory) and presents the findings of three separate G-theory studies. For writing, the focus was on evaluating the impact on…

The Collis Attitudes toward Computers Survey, developed by B. Collis (1984) for a secondary school population, was tested with 305 college students attending the University of Winnipeg (Canada). Results support the reliability and generalizability of the survey and identified one dominant factor--personal interest and enjoyment of computers. (SLD)

Dependability of the Children's Depression Inventory (CDI) was studied using both generalizability and classical test score analyses with a sample of 164 elementary school students. Results suggest that sources of error variance interact to decrease dependability of CDI scores. Depression in children might be better assessed through multiple…

Two frame-of-reference rater training approaches were compared for effects on reliability and accuracy of cutoff scores generated by 21 raters using Angoff methods on tests taken by 155 undergraduates. Both approaches result in higher interrater reliability and more accuracy than does a non-frame-of-reference method. (SLD)

Studied teacher certainty in 1,153 Norwegian teachers in elementary and junior high schools using confirmatory factor analysis and generalizability theory. Results indicate a good fit for a model that operationalizes teacher certainty as a second order latent variable with three first order latent variables: teacher's perceived didactic certainty,…

Purpose: To examine the validity of using scores from a clinical evaluation form as an assessment of clinical competence. Method: Investigators collected a longitudinal clinical skills assessment database that included scores reflecting performance on standardized patient interactions, case-based learning performance, scores on multiple-choice…

A pilot study was conducted to evaluate and improve the rating procedure proposed for use in a research effort designed to assess the essay writing ability of college sophomores. Generalizability theory and the Many-Facet Rasch Model were each used to (a) estimate potential sources of error in the rating, (b) to obtain reliability estimates, and…