Even when raters demonstrate agreement in the use of a measure, limited score variability or violation of often-ignored statistical assumptions can result in lower reliability estimates than intuitively expected. This article uses data drawn from two randomized controlled trials of schema therapy and cognitive behavioral therapy for the treatment…

This brief provides tips for forming a team of staff and consultants with the needed expertise to make key measurement decisions that will ensure high-quality data for answering the study's research questions. The brief outlines the main responsibilities of measurement team members. It also describes typical measurement tasks and discusses…

This is the first book to gather and address what we have learned about the impacts and challenges of data-intensive teacher evaluation systems--a defining characteristic of the current education policy landscape. Expert researchers and practitioners speak to what we know (and what remains to be known) about evaluation measures themselves, the…

This guide is designed to help researchers make sure that their research reports include enough information about study measures so that readers can assess the quality of the study's methods and results. The guide also provides examples of write-ups about measures and suggests resources for learning more about these topics. The guide assumes…

It has been suggested that both internal (e.g., acceptability) and external (e.g., feasibility) factors should be taken under consideration in order to fully understand children's usage of interventions designed to improve their behavior. The purpose of this study was to initiate development of a student self-report measure (Children's Usage…

The consensual assessment technique (CAT) is a measurement tool for creativity research in which appropriate experts evaluate creative products [Amabile, T. M. (1996). "Creativity in context: Update to the social psychology of creativity." Boulder, CO: Westview]. However, the CAT is hampered by the time-consuming nature of the products (asking…

Researchers using structural equation modeling (SEM) aspire to learn about the world by seeking models with causal specifications that match the causal forces extant in the world. This quest for a model matching existing worldly causal forces constitutes an ontology that orients, or perhaps reorients, thinking about measurement validity. This…

Four experienced stuttering researchers viewed videodisks of spontaneous speech from chronic stutterers and attempted to locate the precise onset and offset of individual stuttering events. Results showed interjudge disagreements that challenge the reliability and validity of onset and offset judgments. Highly agreed stuttering events were…

Too many researchers speak of "the reliability of the test," thus indicating their basic misunderstanding of reliability. This paper explains classical reliability and the score features that influence coefficient alpha. It explains when coefficient alpha can be negative, even though it is conceptually a variance-accounted-for statistic.…

Early identification of social-emotional behavioral problems in infants and preschoolers is critical. Nine parent-report and caregiver/teacher-report instruments measuring preschool social-emotional behavioral problems and strengths are reviewed. Advantages to the use of parent-report and caregiver/teacher-report instruments are that they are easy…

The purpose of this paper is to compare the precision of direct and indirect measures of writing assessment using the test information functions from a graded response Item Response Theory (IRT) model. Subjects were 192 sophomore English students from a parochial high school in Wisconsin. Both direct and indirect measures of writing ability were…

This study, involving five experienced and trained speech language pathologists, categorized the elicited imitations of five profoundly and five severely prelingually hearing-impaired subjects using the Phonetic Level Evaluation. Failure to obtain acceptably high levels of reliability suggests that this measure may not yet be an accurate and…

The purpose of this paper is to discuss ways in which developmental psychology suffers from the lack of an appropriate technology of measurement and statistical analysis. The paper begins by noting that developmental psychology is the study of change; that individuals develop through a succession of "stages" which are separated by…

Cason and Cason's model of performance rating was used to determine the extent to which variation in reviewer standards affected the reliability and validity of the program review process used to select papers for inclusion in the annual program. Data analyzed were the overall recommendation for acceptance and ratings on seven quality criteria…

Generalizability theory, which subsumes classical measurement theory as a special case, provides a general model for estimating the reliability of observational rating data by estimating the variance components of the measurement design. Research data from the "Heart Smart" health intervention program were analyzed as a heuristic tool.…