Differences in rater judgments that are systematically related to construct-irrelevant characteristics threaten the fairness of rater-mediated writing assessments. Accordingly, it is essential that researchers and practitioners examine the degree to which the psychometric quality of rater judgments is comparable across test-taker subgroups.…

Mokken scale analysis is a probabilistic nonparametric approach that offers statistical and graphical tools for evaluating the quality of social science measurement without placing potentially inappropriate restrictions on the structure of a data set. In particular, Mokken scaling provides a useful method for evaluating important measurement…

Raters of Georgia's (USA) state-mandated college-level writing exam, which is intended to ensure a minimal university-level writing competency, are trained to grade holistically when assessing these exams. A guiding principle in holistic grading is to not focus exclusively on any one aspect of writing but rather to give equal weight to style,…

Used an analytic rubric to score 120 writing samples from Georgia's 11th grade writing assessment. Raters augmented scores by adding a "+" or "-" to the score. Results indicate that this method of augmentation tends to improve most indices of interrater reliability, although the percentage of exact and adjacent agreement…

States are increasingly turning to standardized testing to hold colleges accountable for student outcomes. Currently, about half of the states require public colleges to conduct some type of assessment or accountability measure. Assessments generally fall into two categories: high-stakes tests, which may affect the progress of individual students,…

Rater errors (rater severity, halo effect, central tendency, and restriction of range) are described, and criteria are presented for evaluating rating quality based on a many-faceted Rasch (FACETS) model. Ratings of 264 compositions from the Eighth Grade Writing Test in Georgia by 15 raters illustrate the discussion. (SLD)

The effects of presenting a choice of writing tasks on the quality of essays produced by eleventh graders were studied with 34,200 students in Georgia. The choice condition had no substantive effect on the quality of essays, but race, gender, and the writing task variable did. (SLD)

The influences of writing tasks and gender on the quality of student writing of black and white eighth graders were examined. Data from statewide writing assessments of 170,899 Georgia students indicated both writing tasks and student characteristics were significant predictors of writing quality. There were both racial and gender differences. (SM)