We propose a new framework for conceptualizing measures of instructed second language (L2) pronunciation performance according to three sets of parameters: (a) the constructs (focused on global vs. specific aspects of pronunciation), (b) the scoring method (human raters vs. acoustic analyses), and (c) the type of knowledge elicited (controlled vs.…

We systematically evaluated the peer-reviewed Rorschach validity literature for the 65 main variables in the popular Comprehensive System (CS). Across 53 meta-analyses examining variables against externally assessed criteria (e.g., observer ratings, psychiatric diagnosis), the mean validity was r = 0.27 (k = 770) as compared to r = 0.08 (k = 386)…

This is a mixed methods investigation of consistency in PhD examination. At its core is the quantification of the content and conceptual analysis of examiner reports for 804 Australian theses. First, the level of consistency between what examiners say in their reports and the recommendation they provide for a thesis is explored, followed by an…

This study examined the effects of training on peer assessment and comments provided regarding oral presentations in EFL (English as a Foreign Language) classrooms. In Study 1, both the treatment and control groups received instruction on skill aspects, but only the treatment group was given an additional 40-minute training on how to rate…

The manner in which results and methods are reported influences the ability of the synthesis of prior studies for planning new evaluations. Confidence ratings, coding conventions, and supplemental evidence can partially overcome the difficulties. Planners must acknowledge the influence of their own judgement in using prior research. (Author)

The purpose of this concluding chapter has been to stimulate creative thinking about how to use cases for evaluation teaching and training. The preparation of professional evaluators presents special challenges. Imaginative new teaching and training resources are appearing, as evidenced by the special sections on evaluation teaching that now…

Examines the extent to which the effect of gender of a target person on evaluators' judgments reflects evaluators' tendencies not to use the evaluee's gender because they assume case information is more valuable. Results with 144 female and 92 male college students in 2 studies demonstrate that evaluators do use gender stereotypes. (SLD)

The implementation of research design in studies on substance abuse education is discussed in this meta-analysis. Findings indicate that the quality of research design is constrained by negative feedback from evaluation stakeholders. This finding is applied to an evaluation of the Here's Looking at You, 2000 drug education program currently…

Eight articles discuss the teaching of evaluation methods through the process of performing evaluations. Four articles concentrate on evaluation as taught in higher education and professional education, and four articles deal with the four-state East Central Acquired Immune Deficiency Syndrome Education and Training Center Project. (SLD)