Generalizability theory (GT) offers a useful framework for estimating the reliability of a measure while accounting for multiple sources of error variance. The purpose of this study was to use GT to examine multiple sources of variance in and the reliability of school-level teacher and high school student behaviors as observed using the tool,…

The ontology of the field of international education is described and analysed in terms of singular, regional and generic modes of pedagogized knowledge. In contrast to singular and regional modes of knowledge which synthesize introjection (orientation onto themselves) and projection (orientation towards external contingencies), generic modes…

This study examined the number of writing samples needed to obtain a reliable estimate of young struggling writers' capabilities. It further assessed if performance in one genre was reflective of performance in other genres for these children. Second- and third-grade students (81 boys, 56 girls), who were identified as struggling writers in need…

One method to investigate classroom quality is for a person to observe what is happening in the classroom. However, this method raises practical and technical concerns such as how many observations to collect, when to collect these observations and who should collect these observations. The purpose of this study is to provide empirical evidence to…

Problem Statement: Reliability, which refers to the degree to which measurement results are free from measurement errors, as well as its estimation, is an important issue in psychometrics. Several methods for estimating reliability have been suggested by various theories in the field of psychometrics. One of these theories is the generalizability…

Measuring school and classroom environments has become central in a nation-wide effort to develop comprehensive programs that measure teacher quality and teacher effectiveness. Formulating successful programs necessitates accurate and reliable methods for measuring these environmental variables. This paper uses a generalizability theory framework…

Reading-to-write (RTW) tasks are becoming increasingly popular and have already been used in several high-stakes English proficiency exams, either replacing or complementing a prompt-based essay test. However, it is still not clear that what accounts for successful or unsuccessful performance on an integrated reading-writing task is owing to the…

Starting in 2012-13, researchers have worked in partnership with Chicago Public Schools (CPS) and the Chicago Teachers Union (CTU) to study implementation of Chicago's new teacher evaluation system: Recognizing Educators Advancing Chicago's Students (REACH). This paper presents findings as well as experiences from the collaboration with CPS and…

There are situations within middle school settings where measurements of students and teachers are used for high-stakes decisions. For example, student performance is used as an indicator of teacher quality or determines student eligibility for particular types of support services. Given the high-stakes nature of these types of assessments,…

The current paper on writing assessment surveys the literature on the reliability and validity of essay tests. The paper aims to examine the two concepts in relationship with essay testing as well as to provide a snapshot of the current understandings of the reliability and validity of essay tests as drawn in recent research studies. Bearing in…

Policymakers and school administrators have embraced value-added models of teacher effectiveness as tools for educational improvement. Teacher value-added estimates may be viewed as complicated scores of a certain kind. This suggests using a test validation model to examine their reliability and validity. Validation begins with an interpretive…

Objective: Crits-Christoph, Connolly Gibbons, Hamilton, Ring-Kurtz, and Gallop (2011) used generalizability theory to critique the measurement of the therapeutic alliance in psychotherapy research, showing that the dependability of alliance scores may be quite low, which in turn can lead to attenuated alliance-outcome correlation estimates. Method…

Student-teacher interactions are dynamic relationships that change and evolve over the course of a school year. Measuring classroom quality through observations that focus on these interactions presents challenges when observations are conducted throughout the school year. Variability in observed scores could reflect true changes in the quality of…

This technical report is one in a series of five describing the reliability (test/retest/and alternate form) and G-Theory/D-Study research on the easy CBM reading measures, grades 1-5. Data were gathered in the spring 2011 from a convenience sample of students nested within classrooms at a medium-sized school district in the Pacific Northwest. Due…

This technical report is one in a series of five describing the reliability (test/retest an alternate form) and G-Theory/D-Study on the easyCBM reading measures, grades 1-5. Data were gathered in the spring of 2011 from the convenience sample of students nested within classrooms at a medium-sized school district in the Pacific Northwest. Due to…