Even when raters demonstrate agreement in the use of a measure, limited score variability or violation of often-ignored statistical assumptions can result in lower reliability estimates than intuitively expected. This article uses data drawn from two randomized controlled trials of schema therapy and cognitive behavioral therapy for the treatment…

This book is relevant for language testers, listening researchers, and oral proficiency teachers, in that it explores four broad themes related to the assessment of L2 listening ability: the use of authentic, real-world spoken texts; the effects of different speech varieties of listening inputs; the use of audio-visual texts; and assessing…

This guide is designed to help researchers make sure that their research reports include enough information about study measures so that readers can assess the quality of the study's methods and results. The guide also provides examples of write-ups about measures and suggests resources for learning more about these topics. The guide assumes…

Background: Youth programs often rely on self-reported data without clear evidence as to the accuracy of these reports. Although the validity of self-reporting has been confirmed among some high school and college age students, one area that is absent from extant literature is a serious investigation among younger children. Moreover, there is…

As the education reform movement increasingly focuses on teachers and teaching, educators, policy-makers, and researchers need valid and reliable measures that can be used to evaluate individual teachers, provide guidance for improving teaching performance, and support research in ways that advance instruction and classroom dialog and practice. A…

The most commonly used measures of locus of control are Rotter's Internality-Externality Scale (I-E) and Nowicki and Strickland's Internality-Externality Scale (NSIE). A reliability generalization study is conducted to explore variability in I-E and NSIE score reliability. Studies are coded for aspects of the scales used (number of response…

The purpose of the present study was to meta-analytically investigate the score reliability for the Satisfaction With Life Scale. Four-hundred and sixteen articles using the measure were located through electronic database searches and then separated to identify studies which had calculated reliability estimates from their own data. Sixty-two…

The Correct Information Unity analysis for measuring the communicative information and efficiency of connected speech was applied to the naturally occurring conversation of a person with moderate aphasia. Results indicated low intrarater and interrater reliability although reliability of word counts was good. Most rater disagreements resulted from…

Too many researchers speak of "the reliability of the test," thus indicating their basic misunderstanding of reliability. This paper explains classical reliability and the score features that influence coefficient alpha. It explains when coefficient alpha can be negative, even though it is conceptually a variance-accounted-for statistic.…

This study is an attempt at a cohesive characterization of the concept of essay reliability. As such, it takes as a basic premise that previous and current practices in reporting reliability estimates for essay tests have certain shortcomings. The study provides an analysis of these shortcomings--partly to encourage a fuller understanding of the…

This paper isolates and examines separately three distinct sources of error in essay scores: lack of agreement between raters; inconsistencies in performance within mode of discourse, and inconsistencies in performance between modes of discourse. Essay prompts in the Iowa Tests of Basic Skills (ITBS) Writing Supplement were designed to assess…

Techniques derived from item response theory are useful for estimating the reliability of test classification above and below the cutting score. Test developers can construct a test whose information is peaked in the region of the cutting score; users can select a test which provides the most information in this region. The Cut-Score…

Two methods of 'equating' tests using item response theory (IRT) are compared, one using true scores, the other using the estimated distribution of observed scores. On the data studied, they yield almost indistinguishable results. This is a reassuring result for users of IRT equating methods. (Author)

The importance of using the standard error of measurement (SEm) in determining reliability in test scores is emphasized. The SEm is compared to the hypothetical true score for standardized tests, and procedures for calculation of the SEm are explained. (JDD)

The stability of intelligence test scores over time was examined for the Wechsler Intelligence Scale for Children-Revised (WISC-R). Subjects included 64 children aged 6-16, who had been administered the WISC-R during prior evaluations. These students had been referred because of academic difficulties. One-third of the sample had taken the test…