This paper presents the interim results of a set of studies undertaken to develop a much needed methodology for establishing content validity in domain-referenced achievement tests. The study results are presented in the context of the larger issue of the improvement of test design. School teachers, administrators and graduate students were trained in using the Item Review Scale (IRS), and then served as item raters. The IRS provides structure for judgments of congruence between items and domain specifications. Raters were given a domain specification (elementary level geology) and eight related items to be rated for validity in terms of match. The items were systematically flawed along one or more of the dimensions of the IRS. Results demonstrated high inter-rater reliability on each dimension and the instrument as a whole. (Author/GK)