Validity, reliability, and fairness are three prominent indicators for evaluating the quality of assessment processes. Each of the indicators is most often written about and applied in the context of large-scale assessment. As a result, the technical properties of these indicators make them limited in both their practicality and relevance for…

Rater-mediated assessments are a common methodology for measuring persons, investigating rater behavior, and/or defining latent constructs. The purpose of this article is to provide a pedagogical framework for examining rater variability in the context of rater-mediated assessments using three distinct models. The first model is the observation…

This inquiry is focused on three indicators of the precision of measurement--conditional on fixed values of ?, the latent variable of item response theory (IRT). The indicators that are compared are (1) The traditional, conditional standard errors, s(eX|?) = CSEM; (2) the IRT-based conditional standard errors, s[subscript irt](eX|?)=C[subscript…

Reliability is a measure of consistency. It is the degree to which student results are the same when they take the same test on different occasions, when different scorers score the same item or task, and when different but equivalent tests are taken at the same time or at different times. Reliability is about making sure that different test forms…

Assuming item parameters on a test are known constants, the reliability coefficient for item response theory (IRT) ability estimates is defined for a population of examinees in two different ways: as (a) the product-moment correlation between ability estimates on two parallel forms of a test and (b) the squared correlation between the true…

This paper aims to develop a valid method to assess the key competencies of the exercise physiology profession acquired through work-integrated learning (WIL). In order to develop a competency-based assessment, the key professional tasks needed to be identified and the test designed so students' competency in different tasks and settings could be…

This article addresses two unsolved measurement issues in dynamic problem solving (DPS) research: (a) unsystematic construction of DPS tests making a comparison of results obtained in different studies difficult and (b) use of time-intensive single tasks leading to severe reliability problems. To solve these issues, the MicroDYN approach is…

This paper illustrates that the psychometric properties of scores and scales that are used with mixed-format educational tests can impact the use and interpretation of the scores that are reported to examinees. Psychometric properties that include reliability and conditional standard errors of measurement are considered in this paper. The focus is…

Accurate measures of youth fitness are needed by researchers and practitioners. Evidence of validity and reliability are essential before results of youth fitness tests can be used to make sound decisions. This article describes a three-stage paradigm for validation research and provides guidance for conducting and understanding norm-referenced…

Reviews issues, raised in part because of "Educational and Psychological Measurement" (EPM) policies, regarding "test reliability," which is psychometric terminology, and "score reliability," score-centric terminology. Discusses datametrics and provides a critique of T. Vacha-Haase's proposed meta-analytic reliability generalization via…

Responds to criticisms of some "Educational and Psychological Measurement" policies and the reliability generalization meta-analytic methods of T. Vacha-Haase. Explores consequences of misunderstanding score reliability. (SLD)

Describes the development and assessment of evaluation instruments designed to test student political-citizenship knowledge, skills, and attitudes. The tests are a part of the Improving Citizenship Education Project in Fulton County, Georgia. (Author/CK)

The Test for Creative Thinking-Drawing Production (TCT-DP), its design, concept and evaluation scheme as well as experiences and results of application are described. The test was designed to mirror a more holistic concept of creativity than the mere quantitatively oriented, traditional divergent thinking tests. The specific design using figural…

Evaluates the use of discrimination indexes (or item-total correlation) for examining the reliability of examinations. States this technique has drawbacks and may cause examination validity to be lower. Discusses the idea of discrimination power and why poor students may answer an item correctly. (CMK)

This article describes the database of the National Repository for Instruments and Strategies Used in the Identification and Evaluation of Gifted Programs (University of Virginia). The Scale for the Evaluation of Gifted Identification Instruments is applied to the Kaufman Assessment Battery for Children. A sample bibliographic reference from the…