Development of the Jordan Performance Appraisal System (JPAS) was completed in 1996. This study examined the factor structure of the classroom observation instrument used in the JPAS. Using observed classroom instructional quality ratings of 1220 elementary teachers of Grades 1-6 in the Jordan School District, this study estimated the factor structure of the data and the rater effect on relevant structural parameters. This study also tested for measurement invariance at the within and between levels across teachers of two grade-level groups (a) lower grades: Grades 1-3 and (b) upper grades: Grades 4-6. Factor structure was estimated using complex exploratory factor analysis (EFA) conducted on a subset of the original data. The analysis provided evidence of a three-factor model for the combined groups. The results of multiple confirmatory factor analyses (CFA) conducted using a different subset of the data cross-validated EFA results. Results from multilevel confirmatory factor analysis (MCFA) indicated the three-factor model fit best at both the within and the between levels, and that the intraclass correlation (ICC) was high (0.699), indicating significant rater-level variance. Results from a multilevel multigroup confirmatory factor analysis (MLMG-CFA) indicated that the ICC was not significantly different between groups. Results also indicated configural, metric (weak factorial), and scalar (strong factorial) equivalence between groups. This study provided one of the first examples of how to estimate the impact of cluster-level variables such as rater on grouping variables nested at the within level. It provided an example of how to conduct a multilevel multigroup analysis on count data. It also disproved the assumption that counting classroom teaching behaviors was less subjective than using a categorical rating scale. These results will provide substantial information for future developments made to the classroom observation instrument used in the JPAS. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://bibliotheek.ehb.be:2222/en-US/products/dissertations/individuals.shtml.]