The reliability, long-term stability, and generalizability of student ratings of teacher effectiveness are discussed. The Students' Evaluation of Educational Quality (SEEQ) instrument is examined from these perspectives. The multidimensionality of student response to such evaluation instruments must be recognized. (SLD)

This paper reports the results of generalizability theory (G) analyses done for new writing and speaking tasks for the Test of English as a Foreign Language (TOEFL). For writing, a special focus was placed on evaluating the impact on the reliability of the number of raters (or ratings) per essay (one or two) and the number of tasks (one, two, or…

The generalizability of scores from different scales in performance assessment was studied. First, a concept map of teachers' and raters' perceptions about various scores and scales was constructed using multidimensional scaling analysis. Then, a generalizability study using a random, partially nested design was conducted to analyze the…

With a construct-centered reliability analytical approach the reliability analysis should crystallize the multi-traits or constructs that the test specialists developed to measure from student performance and then estimate the degree of fit between the theoretical expectations of test developers and the performance exhibited by students. This…

Scholars from psychology and education describe how the new rules of measurement work and how they differ from the old rules. The collection provides an integrated and comprehensive reference source concerned with contemporary issues and approaches in testing and measurement. The chapters are: (1) "Issues in the Measurement of Cognitive…

Results and interpretations of the data from a performance examination were compared for four methods of analysis for 74 medical specialty certification candidates: (1) traditional summary statistics; (2) inter-judge correlations; (3) generalizability theory; and (4) the multifaceted Rasch model. Advantages of the Rasch model are outlined. (SLD)

Explains the use of the construct representation and inference functions of indicators used in construct measurements of self-concept. Illustrates validity and reliability issues with data from 2,114 12th-grade vocational-technical students. (SK)

Using results for 1,487 West German eleventh graders from the International Study of Achievement in Written Composition, problems related to interrater and intrarater reliability and generalizability of composition ratings were assessed. Results indicate that numerous writing assignments are required to assure reliability. (TJH)

A method is presented for determining the optimal number of conditions to use in multivariate-multifacet generalizability designs when resource constraints are imposed. A decision maker can determine the number of observations needed to obtain the largest possible generalizability coefficient. The procedure easily applies to the univariate case.…

A conceptual framework and heuristic model for considering the existence, magnitude, and consequences of context effects are presented through an extension of some generalizability theory concepts. Context effects are often misunderstood, and current measurement models have serious limitations for examining them. Their importance needs to be…

This study, which examined the number of minimal-word-pair exemplars necessary for 19 phonologically impaired children (ages 3-6) to meet a generalization criterion, found substantial variability across subjects, though generalization usually occurred following a small number of exemplars. No relationship was found between specific sounds and the…

Four approaches to estimating interrater reliability in early childhood special education research are illustrated and compared: correlation, comparison of means, percentage of agreement, and generalizability theory techniques. Generalizability theory techniques are proposed as a method for estimating the amount of variance attributable to…

This study used survival analysis to overcome problems of principled generalization and individual variability in analysis of the conversational speech of 24 children with speech delay recorded over two years. The derived normalization probabilities were lagged according to the strong delay hypothesis and results converged with previous normative…

In this study, we discuss the use of generative evaluation as an innovative approach in the analysis of qualitative interview data for evaluating simultaneous renewal of Iowa State University's PT3 grant. Goodlad's (1994) simultaneous renewal framework forms the basis of our argument. We focused on the CREATER+ model designed to explain the…

In this study, we discuss the use of generative evaluation as an innovative approach in the analysis of qualitative interview data for evaluating simultaneous renewal of Iowa State University's PT3 grant. Goodlad's (1994) simultaneous renewal framework forms the basis of our argument. We focused on the CREATER+ model designed to explain the…