This paper investigates the adequacy of the National Assessment of Educational Progress (NAEP) for taking into account dissimilarities in students' family, school, and community contexts when reporting test score differences among population groups (i.e., racial and ethnic minorities). This question was addressed by comparing the NAEP to other representative data for grades 8 and 12 from the National Education Longitudinal Survey (NELS) and High School and Beyond (HSB), studies that contain richer social context measures. These analyses show that the NAEP lacks a number of important social context measures and that the equality of some (but by no means all) of NAEP's measures is low because of reliance on student self-reports and other unreliable data sources. These weaknesses of the NAEP have important practical implications. Compared to HSB and NELS, the NAEP usually overestimates the achievement differences between students who come from different population groups but similar social contexts. However, at the secondary school level at which these analyses were conducted, these overestimates reflect primarily the NAEP's lack of important measures rather than its reliance on student self-reports. (Contains 12 figures, 2 tables, and 53 references.) (Author/SLD)