The authors used cross-cultural methodology to examine the demand-withdraw pattern of marital communication. In Western countries, women usually make more demands, whereas men are more likely to withdraw. But the recently advanced marital structure hypothesis suggests that this pattern can be altered by gender roles and beliefs, particularly in…

This is a construct validation study of a second language speaking assessment that reported a language profile based on analytic rating scales and a composite score. The study addressed three key issues: score dependability, convergent/discriminant validity of analytic rating scales and the weighting of analytic ratings in the composite score.…

Statistical invariance procedures provide a way of looking at the generalizability of research results from sample to sample when the research has not been validated by replication. This paper discusses the Procrustean Rotation invariance procedure following a canonical correlation analysis. The computer program RELATE is used to gauge the…

Not infrequently, investigators assume that reliability for groups is greater than reliability for persons, or that the error variance for groups is less than that for persons. Using generalizability theory, it is shown that this "conventional wisdom" is not necessarily true. Examples are provided from the course-evaluation and the…

Replication is important to viable scientific inquiry; results that will not replicate or generalize are of very limited value. Statistical significance enables the researcher to reject or not reject the null hypothesis according to the sample results obtained, but statistical significance does not indicate the probability that results will be…

This paper describes a four-step approach to constructing diagnostic test profiles that provide precise but practical information on students' instructional needs. A test of pronoun use was constructed to represent 32 categories of usage defined by different combinations of five factors in a domain. (Author/LMO)

This paper describes the processes for collecting information about the objectives of a medical school curriculum and the validation studies of those processes. Major revisions were implemented in the curriculum of the first 2 years at the University of Texas Medical Branch (UTMB) in 1998. In the 2000-2001 academic year, student and faculty groups…

In 1994 the State Collaborative on Assessment and Student Standards of the Council of Chief State School Officers began a study to examine the generalizability of performance-based assessments (PBAs) for state-mandated assessment programs. The intent was to examine the major sources of error associated with PBAs and the generalizability and…

A study evaluated and compared four methods of assessing reliability on one discourse analysis procedure--a modified version of Damico's Clinical Discourse Analysis. The methods were Pearson product-moment correlations; interobserver agreement; Cohen's kappa; and generalizability coefficients. The strengths and weaknesses of the methods are…

Performed generalizability analyses of expert ratings and computer-produced scores for a computer-delivered performance assessment of physicians' patient management skills. The two automated scoring systems produced scores for the 200 medical students that were approximately as generalizable as those produced by the four expert raters. (SLD)

This study uses generalizability theory to examine the dependability of anchoring labels of Likert-type scales. Variance components associated with labeling were estimated in two samples using a two-facet random effect generalizability-study design. In one sample, 173 graduate students in education were administered 7 items measuring attitudes…

A Monte Carlo approach is proposed, using the Statistical Analysis System (SAS) programming language, for estimating reliability coefficients in generalizability theory studies. Test scores are generated by a probabilistic model that considers the probability for a person with a given ability score to answer an item with a given difficulty…

As performance-based assessments have gained wider use, there are increasing concerns about their dependability. This study is a synthesis of existing studies regarding the reliability or generalizability of performance assessments. The meta-analysis involves summarizing, examining, and evaluating research findings. Articles on the dependability…

A study on sampling errors of variance components was conducted within the framework of generalizability theory by P. L. Smith (1978). The study used an intuitive approach for solving the problem of how to allocate the number of conditions to different facets in order to produce the most stable estimate of the universe score variance. Optimization…

The need for using invariance procedures to establish the external validity or generalizability of statistical results has been well documented. Invariance analysis is a tool that can be used to establish confidence in the replicability of research findings. Several approaches to invariance analysis are available that are broadly applicable across…