Evaluators use their eyes to see what is there, whether it is intended or not. But they use their test instruments to measure what is intended, whether it is there or not. Evaluators have been broadening their repertoire of instruments for years: curriculum-embedded tests, observer checklists, audiotape recorders, videotape recorders, unobtrusive…

This monograph has grown out of a series of discussions and a six-month period of reading and reflecting on the literature which were initiated by Fritz Mosher's suggestions to the National Assessment of Educational Progress (NAEP) to examine the "standards" question. Conversations with Mosher himself and the staff of NAEP have been most…

The educational technologies of the past quarter century--from teaching machines to minimal competency testing--all share the general purpose of helping educators make better, or more uniform, decisions. The currently favored technique for shaping local decisions is criterion-referenced testing. Some criterion-referenced testers first find out how…

Sacked, Borne man, and Connelly recently discussed several criticisms that are often raised against the use of cognitive tests in selection. One criticism concerns the issue of measurement bias in cognitive ability tests with respect to specific groups in society. Sacked et AL. (2008) stated that "absent additional information, one cannot…

We are pleased that our article prompted this series of four commentaries and that we have this opportunity to respond. We address each in turn. Duckworth and Kaufman and Agars discussed, respectively, two broad issues concerning the validity of selection systems, namely, the expansion of the predictor domain to include noncognitive predictors of…

In "Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art," Rupp and Templin (2008) undertake the ambitious task of providing a thorough portrait of the current state of diagnostic classification models (DCM). In this commentary, the author applauds Rupp and Templin for their…

On virtually all tests these days, there is a score that determines whether a student passes or fails, is proficient or not, or is being educated or left behind. This is the cut score. This author states that, while it might seem cynical, the fairest way to set the cutoff score is to decide in advance how many students a person wants to fail. This…

In this commentary, the author addresses what is referred to as the deterministic input, noisy "and" gate (DINA) model. The author mentions concerns with how this model has been formulated and presented. In particular, the author points out that there is a lack of recognition of the confounding of profiles that generally arises and then discusses…

Rupp and Templin (2008) do a good job at describing the ever expanding landscape of Diagnostic Classification Models (DCM). In many ways, their review article clearly points to some of the questions that need to be answered before DCMs can become part of the psychometric practitioners toolkit. Apart from the issues mentioned in this article that…

In this commentary, the authors discuss some of the issues regarding the use of diagnostic classification models that practitioners should keep in mind. In the authors experience, these issues are not as well known as they should be. The authors then provide recommendations on diagnostic scoring.

Research Assessment Exercise (RAE) usually uses three indicators in assessing research studies. These indicators are rigour, significance, and originality. In Ian Stronach's article, he claimed that RAE seemed to favor rigour over the other two rubrics. In this article, the author provides her arguments to Stronach's assumptions. The author…

On a general level, the objective of diagnostic classifications models (DCMs) lies in a classification of individuals regarding multiple latent skills. In this article, the authors show that this objective can be achieved by multidimensional adaptive testing (MAT) as well. The authors discuss whether or not the restricted applicability of DCMs can…

Some fundamental concepts of criterion referenced test (CRT) reliability are highlighted. Emphasis is given to the procedures for determining reliability of scores for individual pupils because this is an area requiring increased awareness by classroom teachers and practitioners. Reliability issues encountered in the evaluation of instructional…

Teachers should be evaluated chiefly by the results they produce. Those results will not be properly detected through the use of norm-referenced achievement tests (NRT) due to the following major deficits of NRT: (1) the descriptions of what is measured by NRT are far too loose; (2) evaluative expectations are unclear; (3) the necessity for NRT to…

Mainstreamed students should be graded in terms of their growth rather than by comparison with nonhandicapped students. Fair evaluation procedures include determining terminal and enabling objectives; pretesting and functional assessment; monitoring progress; measuring achievement; and transferring the evaluation into a meaningful grade. (SK)