This notebook contains the following conference presentations: "Identifying and Validating Task Lists by Business and Industry for Test/Test Item Development" (Charles Losh); "Conducting a Task Analysis for Competency-Based Test Item Development" (Brenda Hattaway); "Writing and Reviewing Test Items: Multiple Choice,…

This manual is designed to accompany reports prepared by the Pennsylvania Department of Education for the 505 school districts in that state, summarizing results of the Educational Quality Assessment. The normative sample is described as being stratified on two dimensions: district enrollment and wealth. Administrative procedures are summarized.…

Principles of test administration, test validity, and accuracy of measurement underlying interpretation of standardized test scores in educational administration, instruction, and guidance are presented. Types of norm-referenced score transformations, including percentiles, standard scores, and grade equivalents, and of criterion referenced…

A new standard setting method, Mapmark, was recently developed by ACT Inc. in the course of a contract with the National Assessment Governing Board (NAGB) to set achievement levels for the 2005 National Assessment of Educational Progress (NAEP) in Grade 12 mathematics. Mapmark includes elements of the bookmark method (Lewis, Mitzel, & Green, 1996;…

This report evaluates an "effective schools" intervention project, the purpose of which was to improve achievement on proficiency tests in selected Knoxville City Schools (KCS). Six target schools, predominantly serving Knoxville's impoverished areas as identified by Chapter I and census data, were selected to work toward improving…

This manual is designed to accompany reports prepared by the Pennsylvania Department of Education for the 505 school districts in that state, summarizing results of the Educational Quality Assessment. The normative sample is described as being stratified on two dimensions: district enrollment and wealth. Administrative procedures are summarized.…

The purpose of these studies was to develop and demonstrate a model for identifying criterion levels of performance that can be rationally defended as being the best level of performance for a particular instructional task. The specific objective was to identify the score on a cloze test that represents the most desirable level of performance on…

The corpus of descriptive terminology associated with achievement testing has expanded considerably in recent years, in large part due to the heightened interest in absolute and/or direct metrics for interpreting test performance plus the development of more rigorous strategies for specifying test content. Widely prevalent disagreement about…

The scaling of a new assessment is a significant undertaking. The scaling of a new assessment designed as a multiple-level, criterion-referenced assessment is even more so. A Guttman approach to scaling was used with the Work Keys selected-response assessments, Reading for Information and Applied Mathematics. Assessments in development in the Work…

Distinct multiple regression models are discussed, along with issues and problems related to their application to the practical problem of identifying effective schools. Statistical results obtained from the application of each model are discussed. The first model was implemented in 1984, and the second in 1992. In 1984, it was decided to identify…

Results of the Wisconsin Pupil Assessment Program's 1984 test administrations are summarized. A sample of 16,000 public school students were tested. Tests included objective-referenced tests in writing and understanding of economics, and the norm-referenced Comprehensive Tests of Basic Skills (CTBS) test of reading, mathematics, and language.…

Focusing on the use of the Metropolitan Achievement Test (MAT) with Project Follow Through evaluations, this report represents an extended discussion of the appropriateness of standardized tests for evaluation and policy research. First, the development of the MAT and its roots in the Army Alpha intelligence test are described in detail. The next…

A discussion of second language program evaluation focuses on the interpretability of test scores as a criterion in program evaluation. It looks at both test design and research design issues. First, eight method-comparison, program evaluation studies that compare acquisition-based and analysis/practice based methods are described. Acquisition…

In July 1984, the Texas Legislature mandated the criterion-referenced tests designed to assess minimum basic competencies in mathematics, reading, and writing. During the 1985-86 school year, the Texas Education Agency instituted the Texas Educational Assessment of Minimum Skills (TEAMS) program to measure student competencies at grades 1, 3, 5,…

This study explores the use of the modified caution index (MCI) for identifying judges whose patterns of recommendations suggest that their judgments might be based on incomplete information, flawed reasoning, or inattention to their standard-setting tasks. It also examines the effect on test standards and passing rates when the test standards of…