The reliability, sensitivity, and criterion-related validity of concept comparison (CC) ratings and computer-derived multidimensional scaling (MDS) maps were studied as ways of assessing reading comprehension. Fifteen experts (2 district coordinators and 13 reading specialists/special education teachers) were included in this study. Reliability was studied by comparing CC ratings and maps produced independently by 5 teachers reading eight 250-word passages from elementary level science and social studies texts. For the 3 of the 8 passages that had sufficient interrater reliability, instrument validity was assessed with 104 reading disabled junior and senior high school students. A randomized control group design compared CC tasks completed before and after students read related or unrelated passages. Students reading related passages produced post-reading CC scores significantly more closely related to teacher scores than did readers of unrelated passages. Student and expert CC scores were then correlated with student scores on: (1) vocabulary and reading comprehension sections from norm-referenced reading tests; and (2) maze tests, multiple-choice questions, and oral reading fluency performance, all based on the passages. These three measures were substantially related to post-reading CC scores, but not to pre-reading scores. Standardized test scores were not significantly related to pre-reading or post-reading CC scores. Reliability and validity results support further research on the use of CC tasks and MDS maps for assessing reading comprehension for older disabled readers. Nine data tables, six figures, and three appendices providing study results are included. A 132-item list of references is provided. (Author/SLD)