Across the field of educational assessment the case has been made for alternatives to the multiple-choice item type. Most of the alternative types of items require a subjective evaluation by a rater. The reliability of this subjective rating is a key component of these types of alternative items. In this paper, measures of reliability are examined, including measures of the reliability between raters within a year, the reliability of groups of raters between years, and other parameters. Measures examined are: (1) percent exact agreement; (2) Cohen's kappa statistic; (3) intraclass correlation coefficient; and (4) minimum variance unbiased estimator of intraclass correlation. These issues have been important in the National Assessment of Educational Progress (NAEP), and are addressed in this paper through the use of NAEP writing data from 1984 through 1990. Eight data tables and 16 references are included. (Author/SLD)