This article addresses the capacity of high stakes tests to measure the most significant kinds of learning. It begins by examining a set of philosophical arguments pertaining to construct validity and alleged conceptual obstacles to attributing specific knowledge and skills to learners. The arguments invoke philosophical doctrines of holism and radical interpretation and the theory of situated learning, and they are found to be unsound. The article goes on to examine the difficulties involved in combining adequate validity and reliability in one test. The literature on test item formats is brought to bear on the potential validity of multiple-choice items, and the rater reliability of constructed-response items is addressed through discussion of the methods used by the Educational Testing Service (USA) and a summary report of alternative methods developed by the author and others in cooperation with the California Golden State Examination. (Contains 5 notes.)