New strategies for evaluation criterion referenced measures (CRM) are discussed. These strategies examine the following issues: (1) the use of normed referenced measures (NRM) as CRM and then estimating the reliability and validity of such measures in terms of variance from an arbitrarily specified criterion score, (2) estimation of the…

A distinction is made between reliability of measurement and reliability of classification; the "criterion-referenced reliability coefficient" describes the former. Application of this coefficient to the probability distribution of possible scores for a single student yields a meaningful way to describe the reliability of a single score. (Author)

The use of cut-off scores with criterion referenced tests is defended in this response to two papers by Gene Glass and Nancy Burton. Suggestions for setting cut-off scores are made. (JKS)

A well-constructed criterion-referenced test has an unambiguous descriptive scheme, an adequate number of items, a sufficiently limited focus, reliability, validity, and comparative data. (Author/MLF)

This study examined the test-retest reliability and criterion validity of basal mastery tests of three commercial reading series. Results indicated that reliability and validity of the test varied among and within instruments. Implications for developing and using basal mastery tests are discussed. (Author/MT)

Investigated the reliability of the McCarthy Screening Test (MST) from a criterion-referenced standpoint, using a sample of 1,323 children. Suggested that dependability indices be adopted for clinical instruments such as the MST. (JAC)

Much of the content was presented at the TESOL Convention, New Orleans, Louisiana, March 1971, and is derived from the author's dissertation Experimental Steps to Ascertain Reliability of Diagnostic Tests in English as a Second Language" (Columbia University). (VM)

For some situations the beta-binomial distribution might be used to describe the marginal distribution of test scores for a particular population of examinees. Several different methods of approximating the maximum likelihood estimate were investigated, and it was found that the Newton-Raphson method should be used when it yields admissable…

This paper evaluates the magnitude of the total error in estimates of the difference between an examinee's domain score and the cutoff score. An observed score based on a random sample of items from the domain, and an estimated cutoff score derived from a judgmental standard setting procedure are assumed. The work of Brennan and Lockwood (1980) is…

Criterion-referenced evaluation (CRE) describes achievement in performance terms, whereas norm-referenced evaluation (NRE) compares the performance of one individual to that of others with respect to a given evaluation instrument. Vocational educators who base their programs on behaviorism commonly evaluate student performance from a CRE…

The validity and dependability of functional competency tests for adults are examined as they relate to the information needs of instructional decision makers. Test data from the Adult Performance Level (APL) Program (funded by the U.S. Office of Education at the University of Texas at Austin) is used to illustrate key points. In the discussion of…

Comparisons of corresponding values of phi and kappa coefficients were made for 270 instances of data generated by a Monte Carlo technique to simulate a test-retest situation. Data were generated for distributions with the same mean but three different levels of standard deviation, standard error of measurement and cutting score. Ten samples of…

One aim of criterion-referenced testing is to classify an examinee without reference to a norm group; therefore, any statements about the dependability of such classification ought to be group-independent also. A population-independent index is proposed in terms of the probability of incorrect classification near the cutoff true score. The…

The reliability of a criterion referenced test was defined as a measure of the degree to which the test discriminates between an individual's level of performance and a predetermined criterion level. The variances of observed and true scores were defined as the squared deviation of the score from the criterion. Based on these definitions and the…

An attempt is made to explore the use of subjective probabilities in the analysis of item data, especially criterion-referenced item data. Two assumptions are implicit: (1) one wants to obtain a maximum amount of information with respect to an item using a minimum number of subjects; and (2) once the item is validated, it may well be administered…