When the National Assessment of Educational Progress (NAEP) was designed 20 years ago, comparisons among individual states or localities were not deemed desirable. Today, this lack of information to allow comparison is judged to be a serious weakness of the NAEP, and ways to allow comparisons are actively sought. The focus of this paper is to review issues concerning the definition of the domain of content to be covered in the NAEP and the relationship of the definition and score reporting systems to the validity of inferences that are based on state-by-state comparisons. This validity is the most fundamental issue in development of the NAEP. Essential issues in the content domain include: (1) breadth of coverage and the match with what is taught and tested; and (2) the number and specificity of scores. Ideally, the domain for assessment would include separate measures of the full range of outcomes that are considered important by any of the states. Despite the desirability of having multiple scores for purposes of identifying strengths and weaknesses, global scores will still need to be produced, because multiple scores would be too overwhelming and because there is a desire for a score card. It is argued that the ability to disaggregate results to specific content areas should be retained. A 37-item list of references is included. (SLD)