The purpose of this study was to determine the degree to which inter-rater agreement in classroom observation of teachers is affected by using a system where each scale point is defined by a descriptor particularly created for the quality being rated. Thirty-three subjects enrolled in two Teacher Performance Assessment Instruments training sessions were involved in the study. About two-thirds of the trained observers were able to choose the same scale point when descriptors were available. Rates of agreement were increased from about 45 percent to about 70 percent when descriptors were available and observers trained in their use. Differences in mean ratings suggested that different qualities were assessed when no descriptors were used. (JD)