Studies on automatic scoring systems in writing assessments have also evaluated the relationship between human and machine scores for the reliability of automated essay scoring systems. This study investigated the magnitudes of indices for inter-rater agreement and discrepancy, especially regarding human and machine scoring, in writing assessment.…

Teacher evaluation's relationship with instructional improvement is under-theorized in the literature. To address this gap, this paper uses a conceptual framework rooted in human, social, and material capital to analyze and synthesize findings from research conducted since 2009 on whether and under what conditions teacher evaluation stimulates…

We propose a new framework for conceptualizing measures of instructed second language (L2) pronunciation performance according to three sets of parameters: (a) the constructs (focused on global vs. specific aspects of pronunciation), (b) the scoring method (human raters vs. acoustic analyses), and (c) the type of knowledge elicited (controlled vs.…

We systematically evaluated the peer-reviewed Rorschach validity literature for the 65 main variables in the popular Comprehensive System (CS). Across 53 meta-analyses examining variables against externally assessed criteria (e.g., observer ratings, psychiatric diagnosis), the mean validity was r = 0.27 (k = 770) as compared to r = 0.08 (k = 386)…

This article presents a study conducted by Cornelia Yarbrough and Jennifer Whitaker titled "Analysis of Reviewer Comments About Quantitative Manuscripts Accepted by the "Journal of Research in Music Education"." The study aims to analyze reviewers' comments for quantitative manuscripts with regard to the following categories: section discussed…

On the basis of a multistage exploration of evaluation texts, electronic searches, and nominations from the field and from managing social science, the author concludes there is little research literature on managing evaluation studies, evaluators and other workers, and evaluation units. The discussion explores what this limited literature tells…

This is a mixed methods investigation of consistency in PhD examination. At its core is the quantification of the content and conceptual analysis of examiner reports for 804 Australian theses. First, the level of consistency between what examiners say in their reports and the recommendation they provide for a thesis is explored, followed by an…

Four types of analysis are commonly applied to data from structured Rater [times] Ratee designs. These types are characterized by the unit of analysis, which is either raters or ratees, and by the design used, which is either between-units or within-unit design. The 4 types of analysis are quite different, and therefore they give rise to effect…

This study examined the effects of training on peer assessment and comments provided regarding oral presentations in EFL (English as a Foreign Language) classrooms. In Study 1, both the treatment and control groups received instruction on skill aspects, but only the treatment group was given an additional 40-minute training on how to rate…

The article suggests supplementing the social problem study groups and data synthesis approaches to building knowledge with emphasis on synthesizing reviews. Such reviews would strengthen the evaluation profession and promote utilization of the knowledge base generated by evaluators. (DWH)

The manner in which results and methods are reported influences the ability of the synthesis of prior studies for planning new evaluations. Confidence ratings, coding conventions, and supplemental evidence can partially overcome the difficulties. Planners must acknowledge the influence of their own judgement in using prior research. (Author)

Educational software makers are often rebuffed by educational authorities, whose endorsements could lead to governmental stamps of approval, and thus explosive sales. But they usually get warmer receptions in the offices of the nation's school superintendents, who are, after all, their primary customers. The system was not supposed to work this…

The purpose of this concluding chapter has been to stimulate creative thinking about how to use cases for evaluation teaching and training. The preparation of professional evaluators presents special challenges. Imaginative new teaching and training resources are appearing, as evidenced by the special sections on evaluation teaching that now…

Examines the extent to which the effect of gender of a target person on evaluators' judgments reflects evaluators' tendencies not to use the evaluee's gender because they assume case information is more valuable. Results with 144 female and 92 male college students in 2 studies demonstrate that evaluators do use gender stereotypes. (SLD)

Eight articles discuss the teaching of evaluation methods through the process of performing evaluations. Four articles concentrate on evaluation as taught in higher education and professional education, and four articles deal with the four-state East Central Acquired Immune Deficiency Syndrome Education and Training Center Project. (SLD)