Generalizability theory (GT) offers a useful framework for estimating the reliability of a measure while accounting for multiple sources of error variance. The purpose of this study was to use GT to examine multiple sources of variance in and the reliability of school-level teacher and high school student behaviors as observed using the tool, Assessing School Settings: Interactions of Students and Teachers (ASSIST). The study took place in the context of a high school randomized controlled trial (Maryland Safe and Supportive Schools [MDS3]--58 high school randomized controlled trials funded by the U.S. Department of Education and the William T. Grant Foundation) of Positive Behavioral Interventions and Supports (PBIS) (Sugai & Horner 2006). PBIS is a school-based decision-making framework designed to enhance students' social and academic functioning. Baseline data (collected prior to intervention) from six Maryland high schools provided the analytic sample for the study. Two raters worked in tandem to observe 25 classrooms/occasions in each of the six high schools using the ASSIST, resulting in a partially crossed design in which observations were nested within schools and crossed by rater. School interactions were assessed using the ASSIST, an observational measure of teachers' behavior management strategies and student behavior. Generalizability studies (G-studies) and decision studies (D-studies) were conducted using GENOVA software. Variations in the number of raters used to conduct observations and the number of observations occurring within a school had direct implications for the reliability of ASSIST tally items. Results indicate that raters can be trained to observe selected behaviors with high levels of agreement, and that studies using the ASSIST may be well-positioned to capture school-level differences for the behaviors of teacher proactive behavior expectations, teacher reactive behavior management, student disruption, and student profanity. This study demonstrates GT as a useful tool when evaluating and planning for the use of observational measures in large-scale program evaluations. Tables and figures are appended.