The purpose of this study was to compare the relative magnitude of the variance components and generalizability coefficients derived from the Teacher Performance Assessment Instruments (TPAI) data using two different methods of data collection: (1) occasions when observers were in the classroom for simultaneous observation and (2) occasions when the observers made independent observations. The two data sets used in the analysis were collected as part of the field testing of the revised TPAI, and they consisted of two different sets of observations of 23 teachers. One set consisted of observations by two different observers present in the teachers' classrooms on the same day; the other set consisted of observations of these same teachers by two different observers on different days. Observations from each of these data sets were scored using the criteria anticipated when the instruments are used in certification, including the use of essential descriptors. Satisfactory values for the generalizability coefficient were obtained when two observers visited the classroom on separate days, but a single day of observation did not provide sufficiently reliable data to differentiate teachers. (LMO)