This study examines score variations and differences in the reliability of ratings between English-as-a-second-language (ESL) and native English (NE) authored papers in a graduate course. Generalizability (G-) theory was used as a framework for analysis because it is powerful in detecting rater variability and the relative contributions of multiple sources of error. The results indicate that ESL papers received consistently lower scores than NE papers. The G-coefficients for ESL and NE papers were considerably different revealing concern about the reliability of ratings of ESL papers. The significant increase in the number of ESL students pursuing graduate degrees in North American institutions warrants further research to determine the extent to which consistency differences affect the validity of the assessment of ESL students' writing and to identify ways to alleviate these differences. (Contains 5 tables and 1 footnote.)