In the fall of 2014, the University at Buffalo created a new universal Student Evaluation of Teaching (SET). The purpose of the present study was to establish the construct validity of SET items. Rasch analyses of data from 7 semesters (N=203,194 students) revealed problems with item fit indices and threshold distances. Changes to items and…

Based on the Job Demands-Resources model, the study developed and validated an instrument that measures physical education teachers' job demands/resources perception. Expert review established content validity with the average item rating of 3.6/5.0. Construct validity and reliability were determined with a teacher sample (n=193). Exploratory…

Purposes: This study uses think-alouds and cognitive interviews to provide validity evidence for an online formative assessment--i-Ready Standards Mastery (iSM) mini-assessments--which involves a heavy use of innovative items. iSM mini-assessments are intended to help teachers determine student understanding of each of the on-grade-level Common…

In this paper, we describe the initial stage of reliability and validity testing for the Mathematics Education of English Learners Scale (MEELS), which is designed to measure preservice teachers' beliefs about the mathematics education of English learners. To address the content validity, we consulted with experts within the field of mathematics…

Developed for the Armed Services Vocational Aptitude Battery (ASVAB) Career Exploration Program, the Find Your Interests (FYI) inventory was designed to help students learn about their career-related interests. The FYI is a 90-item interest inventory based on Holland's (1973, 1985, 1997) widely accepted theory and taxonomy of career choice. The…

Examines testing practices in: (1) the past, in which the traditional paradigm left little room for cognitive psychology principles; (2) the present, in which testing research is enhanced by principles of cognitive psychology; and (3) the future, in which the potential of cognitive psychology should be fully realized through item design.…

Assessing the construct relevance of mental test results continues to present many challenges, and it has proven to be particularly difficult to assess the construct relevance of verbal items. This study was conducted to gain a better understanding of the conceptual sources of verbal item difficulty using a unique approach that integrates…

A study examined the construct validity of second language listening comprehension tests that use different types of texts by determining to what degree the text's listenability facilitates or hinders comprehension. Three hypotheses were tested: (1) texts containing more listenable features, thus closer to the oral end of the oral/literate…

The purpose of this paper is to clarify the seemingly discrepant views of test theorists and test developers about terminology related to the evaluation of test content. The origin and evolution of the concept of content validity are traced, and the concept is reformulated in a way that emphasizes the notion that content domain definition,…

This study was conducted to investigate the construct validity of scores on the Personal Preferences Self-Description Questionnaire (PPSDQ), a measure of Jungian types. Confirmatory factor analysis methods were used to investigate the structures underlying PPSDQ responses of 641 university students. The model fit statistics were generally…

By definition, differential item functioning (DIF) refers to unequal probabilities of a correct response to a test item by examinees from two groups when controlled for their ability differences. Simulation results are presented for an attempt to purify a test by separating out multidimensional items under the assumption that the intent of the…

In an address to the National Council on Measurement in Education, R. M. Jaeger (1987) commented that there appears to be a fundamental difference in measurement philosophy between those on the two sides of the debate over the Rasch model. Jaeger's observations are explicated by contrasting the views on measurement of B. D. Wright and E. F.…

A study examined the relationship of native language and level of English proficiency to the structure of the Test of English as a Foreign Language (TOEFL). Using all of the information provided by various responses to the test's items (the four alternatives, omitted, and not reached), the items' interrelations were analyzed by three-way…

The concept of a user-specified validity sector is discussed. The idea of the validity sector combines the work of M. D. Reckase (1986) and R. Shealy and W. Stout (1991). Reckase developed a methodology to represent an item in a multidimensional latent space as a vector. Item vectors are computed using multidimensional item response theory item…

An expanded version of the method of content evaluation proposed by S. G. Sireci and K. F. Giesinger (1992) was evaluated with respect to a national licensure examination and a nationally standardized social studies achievement test. Two groups of 15 subject-matter experts rated the similarity and content relevance of the items. (SLD)