This study is an attempt at a cohesive characterization of the concept of essay reliability. As such, it takes as a basic premise that previous and current practices in reporting reliability estimates for essay tests have certain shortcomings. The study provides an analysis of these shortcomings--partly to encourage a fuller understanding of the…

Comments are made on the review papers presented by six Dutch psychometricians: Ivo Molenaar, Wim van der Linden, Ed Roskam, Arnold Van den Wollenberg, Gideon Mellenbergh, and Dato de Gruijter. Molenaar has embraced a pragmatic viewpoint on Bayesian methods, using both empirical and pure approaches to solve educational research problems. Molenaar…

The purpose of this study was to compare the relative magnitude of the variance components and generalizability coefficients derived from the Teacher Performance Assessment Instruments (TPAI) data using two different methods of data collection: (1) occasions when observers were in the classroom for simultaneous observation and (2) occasions when…

The percentage of students at/above a cut point (PAAC) is one of the most common measures used for reporting school-level performance relative to a proficiency standard (L. Cronbach, N. Bradburn, and D. Horvitz, 1994). The two purposes of this study were to introduce procedures for estimating standard errors for school PAACs under a…

Natural statistical models based on renewal processes for individuals' Behavior Stream are developed. For continuous observation, statistical and psychometric properties are obtained for empirical: (1) rates of behavior; (2) proportions of behavior type; (3) prevalence; and (4) event duration. Results provide a guide for design and analysis. (SLD)

The stability of scores on 2 types of performance assessments, an observed hands-on investigation and a notebook surrogate, was investigated for 29 sixth graders on 2 occasions. Results indicate that student performance and procedures changed and that generalizability across occasions was moderate. Implications for assessment are discussed. (SLD)

In this article, we discuss the history behind efforts to transfer school psychology research into practice and review the literature pertaining to treatment acceptability, participatory action research, organizational change, and generalization programming. We then present a model for the systematic programming of this transfer and propose a…

Issues surrounding accurate assessment of depression in children have received much attention. However, the stability of scores from depression measures has generally been estimated using only classical test score theory, rather than the more powerful generalizability theory. The dependability of scores from the Children's Depression Inventory…

The meta-analysis by A. C. Willig of bilingual education evaluation literature was not successful in comparison with and did not replicate the Baker and de Kanter narrative review. Willig's conclusion that bilingual education is effective is not supported by the analysis. (SLD)

The Bookmark Standard Setting Procedure (Lewis, Mitzel, and Green, 1996) is an item-response-theory-based standard setting method that has been widely implemented by state testing programs. The primary purposes of this study were to: (1) estimate standard errors for cutscores that result from Bookmark standard settings under a generalizability…

The multivariate generalizability of the Role Functioning Scale (RFS), which serves as a replacement for the global ratings of the Social Adjustment Scale, was studied with 396 severely disturbed clients of a public mental-health bureau. Generalizability of single subscales was less than that of the combined subscales. (SLD)

Limitations of the confirmatory factor analysis conducted by H. W. Marsh to demonstrate the multidimensionality of a rating form to assess student evaluation of teaching effectiveness are discussed. A secondary analysis suggests that dimensionality is not stable across solutions and a global factor can be retrieved. (SLD)

Evidence from test results of 3,604 sixth and seventh graders is provided for the generalizability and validity of the Quantitative Understanding: Amplifying Student Achievement and Reasoning (QUASAR) Cognitive Assessment Instrument, which is designed to measure program outcomes and growth in mathematics. (SLD)

The New Standards Project conducted a pilot test of a series of performance-based assessment tasks in mathematics and English language arts at Grades 4 and 8 in the spring of 1993. This paper reports the results of a series of generalizability analyses conducted for a subset of the 1993 pilot study data in mathematics. Generalizability analyses…

This annotated bibliography contains nine items addressing assessment methodology. The titles are: "Performance and Product Evaluation"; "The Critical Incident Technique"; "Constructing Achievement Tests"; "Applying the Assessment Center Method"; "Performance Assessment in Education and Training:…