For classical, norm-referenced test reliability, Cronbach's alpha has been shown to be equal to the mean of all possible split-half Pearson product-moment correlation coefficients, adjusted by the Spearman-Brown prophecy formula. For criterion-referenced test reliability, in an analogous vein, this paper provides the rationale behind, the analysis…

Some fundamental concepts of criterion referenced test (CRT) reliability are highlighted. Emphasis is given to the procedures for determining reliability of scores for individual pupils because this is an area requiring increased awareness by classroom teachers and practitioners. Reliability issues encountered in the evaluation of instructional…

It is proposed that the reliability of criterion-referenced test scores be defined in terms of the consistency of the decision-making process across repeated administrations of the test. (Author/RC)

The kappamax reliability index of domain-referenced tests is defined as the upper bound of kappa when all possibile cutoff scores are considered. Computational procedures for kappamax are described, as well as its approximation for long tests, based on Kuder-Richardson formula 21. The sampling error of kappamax, and the effects of test length and…

An index for the dependability of mastery tests is described. Assumptions necessary for the index and the mathematical development of the index are provided. (Author/JKS)

Two indices for assessing the efficiency of decisions in mastery testing are proposed. The indices are generalizations of the raw agreement index and the kappa index. Empirical examples of these indices are given. (Author/JKS)

Discusses problems in using traditional measures of reliability for criterion-referenced tests (CRTs) and describes two approaches to reliability for CRTs: estimates sensitive to all measures of error; and estimates of consistency in test outcome. Compares the two approaches and proposes recommendations for interpretation and use. (Author/LRW)

The study investigated the feasibility of adapting the coefficient k introduced by Cohen (1960) and elaborated by Swaminathan, Hambleton, and Algina (1974) to an internal consistency estimate for criterion referenced tests in single test administrations. The authors proposed the use of k as an internal consistency estimate by logically dividing…

Single-administration procedures for estimating the coefficient of agreement, a reliability index for criterion referenced tests, were recently developed by Subkoviak. The procedures require a distributional assumption for errors of measurement and an estimate of each examinee's true score. A computer simulation of tests composed of items that…

Within the beta-binomial Bayesian framework, procedures are described for the evaluation of the kappa index of reliability on the basis of one administration of a domain-referenced test. Major factors affecting this index include cutoff score, test score variability and test length. Empirical data which substantiate some theoretical trends deduced…

A number of different reliability coefficients have recently been proposed for tests used to differentiate between groups such as masters and nonmasters. One promising index is the proportion of students in a class that are consistently assigned to the same mastery group across two testings. The present paper proposes a single test administration…

The analysis of variance model for estimating reliability in norm referenced tests is extended to criterion referenced tests. The essential modification is that the criterion or cut-off score is substituted for the population mean. An example and discussion are presented. (JKS)

Classical test theory has been rejected for application to criterion-referenced (CR) tests by most psychometricians due to an expected lack of variance in scores and other difficulties. The present study was conceived to resolve the variance problem and explore the possibility that classical test theory is both appropriate and desirable for some…

Issues related to the evaluation of instructional programs developed under the auspices of the Southwest Educational Development Laboratory are briefly discussed. The Laboratory develops criterion-referenced tests which form an integral part of each instructional program. The importance of examining the reliability and validity of these tests is…

Examined relative efficacy of two short forms of Wechsler Adult Intelligence Scale-Revised (WAIS-R) with respect to accurate subtest profile scatter (N=50). Subtest scores of both split-half Satz-Mogel short form and criterion referenced Modified WAIS-R (WAIS-RM) short form were found to differ significantly from full-length WAIS-R subtest scores.…