The construct validity of the New Standards middle school Science Reference Examination was studied focusing on evidence related to the internal and external structure of the assessment, the reliability of the assessment scores, and the generalizability of the assessment results. Data for 450 students were taken from the field test in spring 1998. Results related to the internal structure of the assessment suggest that although the assessment tasks measured a single common factor, this did not detract from the usefulness of scientific thinking or science concept subscores for instructional purposes. With respect to the external structure of the assessment, moderate correlations between the New Standards total scores, the Stanford Achievement Test (ninth edition), and the Otis-Lennon School Aptitude Test (seventh edition) scores provided evidence that the scores from these assessments rank student performance in similar ways. However, these correlations do not indicate that the assessments are measuring the same construct. For evidence for the reliability of the assessment scores and decisions based on them, the results of generalizability studies imply that reader variance could be made negligible by training readers with well-defined scoring rubrics. The high rates of decision consistency and accuracy at different total score cutpoints provide evidence that the New Standards Science Reference Examination could be used reliably to classify student performance on the basis of a total test score. For subscores, providing one cutpoint with a reference point to meet the standards would be instructionally informative. (Contains 12 tables and 23 references.) (Author/SLD)