The main purpose of the study is to evaluate the qualities of human essay ratings for a large-scale assessment using Rasch measurement theory. Specifically, Many-Facet Rasch Measurement (MFRM) was utilized to examine the rating scale category structure and provide important information about interpretations of ratings in the large-scale performance assessment setting. The preliminary results show that though the raters varied slightly in leniency or stringency, they were able to differentiate the three rating dimensions according to the rubrics and apply the rating scale category appropriately. The reliability indices resulted in sufficient examinee separation of 0.90 from the MFRM models. The results provided not only validity evidence for the essay ratings but also implications for quality assurance of rater training.