The applicability is explored of the Bayesian random-effect analysis of variance (ANOVA) model developed by G. C. Tiao and W. Y. Tan (1966) and a method suggested by H. K. Suen and P. S. Lee (1987) for the generalizability analysis of autocorrelated data. According to Tiao and Tan, if time series data could be described as a first-order…

Khuri's and Satterthwaite's methods of obtaining confidence intervals of variance components are compared. The article discusses that Khuri's method may be applied to obtain confidence intervals for the variance components and other linear functions of the expected mean squares used in generalizability theory. (Author/JAZ)

This article argues that the 1981 work of Carbno presented unwarranted conclusions because its design included an improper operationalization of the object of measurement, given the problems addressed, and because the sample sizes employed were too small. (Author/PN)

Job analysis has played a fundamental role in developing and validating licensure and certification examinations, but research on what constitutes reliable and valid job analysis data is lacking. This paper examines the reliability and validity of job analysis survey results. Generalizability theory and the multi-facet Rasch item response theory…

Two studies, each involving 10 eighth graders,compared the findings from generalizability (G) studies and alternative decision (D) studies for 4 approaches to scoring classroom assessments. In terms of less rater x person variability and higher G and D coefficients, the methods ranked in this order: topic-specific rubric, constrained point,…

Two longitudinal studies involving 29 raters concerning the construct validity, temporal stability, and interrater reliability of the latent common factors underlying subjective assessments by human raters of personality traits in the stumptail macaque and the zebra finch illustrate the use of generalizability analysis to test prespecified…

Presents solutions for the problem of maximizing the generalizability coefficient under a budget constraint. Shows that the Cauchy-Schwarz inequality can be applied to derive optimal continuous solutions for the number of conditions of each facet. Illustrates the formal similarity between optimization problems in survey sampling and…

It is proposed that proportional reduction in loss (PRL) be used as a theoretical basis to derive, justify, and interpret reliability measures to gauge reliability on a zero-to-one scale. This PRL approach simplifies the interpretation of existing measures (e.g., generalizability-theory measures). (SLD)

A preprinted evaluation form and generalizability theory were used to judge the reliability of student grading of their peers' papers. Findings suggest that students can be consistent and fair in their assessments. Student practice in peer evaluation will help develop the management skill of employee evaluation. (JOW)

Two chapters of this issue consider critical multiplism as a research strategy with links to meta analysis and generalizability theory. The unifying perspective it can provide for quantitative and qualitative evaluation is discussed. The third chapter explores meta analysis as a way to improve causal inferences in nonexperimental data. (SLD)

Cross-national differences in the prevalence of psychopathy have been reported. This study examined whether rater effects could account for these differences. Psychopathy was assessed with the Psychopathy Checklist-Revised (PCL-R; R. D. Hare, 1991). Videotapes of 6 Scottish prisoners and 6 Canadian prisoners were rated by 10 Scottish and 10…

This article investigates a group oral test as administered at a university in Japan to find if it is appropriate to use scores for higher stakes decision making. It is one component of an in-house English proficiency test used for placing students, evaluating their progress, and making informed decisions for the development of the English…

Value-added models in education research allow researchers to explore how a wide variety of policies and measured school inputs affect the academic performance of students. Researchers typically quantify the impacts of such interventions in terms of "effect sizes", i.e., the estimated effect of a one standard deviation change in the…

This paper presents a review of the literature on reliability in qualitative studies. Reliability is defined as the extent to which studies can be replicated, using the same methods, and getting the same results. It is the degree to which data are independent of the accidental circumstances of the research. The review includes the following three…

Generalizability theory is based upon analysis of variance (ANOVA) and requires estimation of variance components for the ANOVA design under consideration in order to compute either G (Generalizability) or D (Decision) coefficients. Estimation of variance components has a number of alternative methods available using SAS, BMDP, and ad hoc…