A pilot study was conducted to evaluate and improve the rating procedure proposed for use in a research effort designed to assess the essay writing ability of college sophomores. Generalizability theory and the Many-Facet Rasch Model were each used to (a) estimate potential sources of error in the rating, (b) to obtain reliability estimates, and (c) to make recommendations for improving the rating process. Variance due to Task (writing prompt) and the Person-by-Task interaction were high while the variance attributable to Raters and Occasion was low. Twenty-two percent of the variability in the ratings was unexplained. The common and unique features of generalizability theory and the Many-Facet Rasch Model are described, and the advantages and disadvantages of each are discussed. (Contains 6 figures and 6 tables.)