The 12th annual William H. Angoff Memorial Lecture was presented by Dr. Michael T. Kane, ETS's (Educational Testing Service) Samuel J. Messick Chair in Test Validity and the former Director of Research at the National Conference of Bar Examiners. Dr. Kane argues that it is important for policymakers to recognize the impact of errors of measurement…

In this speech, the author covers a lot of ground. In the first half of his speech, the author gives a brief summary of the last 40 years of the history of language testing, from his perspective. The author reviews these years more or less by decade. Additionally, he discusses the evolution of the profession of language testing during this period,…

Much of the current research concerning reliability emphatically suggests that researchers should gather their own reliability estimates when administering an instrument. It has also been recommended that data with low reliability be discarded. While some data obtained from instruments that originally yielded reliable results may be unreliable, it…

Given the concern for reliability in essay evaluation and the prospect of "error" variance in its absence, methods to promote interrater reliability in the evaluation of written compositions have been developed. These methods reduce variation in the value systems being applied by readers to texts, either by limiting the group of readers…

Some fundamental concepts of criterion referenced test (CRT) reliability are highlighted. Emphasis is given to the procedures for determining reliability of scores for individual pupils because this is an area requiring increased awareness by classroom teachers and practitioners. Reliability issues encountered in the evaluation of instructional…

The program evaluation standards approved by the American National Standards Institute (ANSI) in 1994 that deal with reliability and validity accurately represent contemporary views of the psychometric community with regard to reliability and validity. As such, these standards move the field forward. The ANSI standards recognize that reliability…

This paper explores how facets of the concept "rigor" might be applied to questions about the validity and reliability of research independently of the research modes. The focus of the critical lens could then be on how to assess the contribution of various forms of research rather than on the "paradigm wars" and arguments…

The purpose of this paper is to highlight some psychometric cautions that should be observed when seeking to develop short form versions of tests. Several points are made: (1) score reliability is impacted directly by the characteristics of the sample and testing conditions; (2) sampling error has a direct influence on reliability and factor…

This paper is the transcript of a talk given to those who use test information but who have little technical background in test theory. The concepts of modern test theory are compared with traditional test theory, as well as a probable future test theory. The explanations given are couched within an extended metaphor that allows a full description…

Problems inherent in relying solely on statistical significance testing as a means of data interpretation are reviewed. The biggest problem with statistical significance testing is that researchers have used the results of this testing to ascribe importance or meaning to their studies where such meaning often does not exist. Often researchers…

Arguments are presented for conceptualizing validity within a postmodern approach. Validity, reliability, and generalizability have been a holy trinity of social science research, and standard definitions of validity have been taken from criteria developed for psychometric tests. From a postmodern point of view, validity is sometimes discarded as…

A study examined identification and consistency theory of interface between rhetorical and communication theory, to demonstrate the compatibility of specific principles in Kenneth Burke's theory with those in the work of Fritz Heider and other consistency theorists and to make suggestions toward a Burkeian theory of identification through…

The ability of individuals to make accurate judgments about others is examined and literature on this subject is reviewed. A wide variety of situational factors affects the appraisal of performance. It is generally accepted that the purpose of the appraisal influences the accuracy of the appraiser. The instrumentation, or tools, available to the…

Three areas of concern related to test bias and validity should be considered during the revision of the Standards for Educational and Psychological Tests. The first area concerns the sources and consequences of test bias. Five sources of bias have been identified: numerical bias, role bias, status bias, stereotypic bias, and familiarity bias. The…

Noting that improvement in rater reliability means eliminating differences among raters, this paper discusses ways to assess writing evaluator reliability and methods for achieving higher levels of interrater reliability. After showing that reliability can be improved two ways--by increasing the number of raters or measurements made, and by…