Comprehension tests are often used interchangeably, suggesting an implicit assumption that they are all measuring the same thing. We examine the validity of this assumption by comparing some of the most popular reading comprehension measures used in research and clinical practice in the United States: the Gray Oral Reading Test (GORT), the two assessments (retellings and comprehension questions) from the Qualitative Reading Inventory (QRI), the Woodcock-Johnson Passage Comprehension subtest (WJPC), and the Reading Comprehension test from the Peabody Individual Achievement Test (PIAT). Modest intercorrelations among the tests suggested that they were measuring different skills. Regression analyses showed that decoding, not listening comprehension, accounts for most of the variance in both the PIAT and the WJPC; the reverse holds for the GORT and both QRI measures. Large developmental differences in what the tests measure were found for the PIAT and the WJPC, but not the other tests, both when development was measured by chronological age and by word reading ability. We discuss the serious implications for research and clinical practice of having different comprehension tests measure different skills and of having the same test assess different skills depending on developmental level. (Contains 6 tables, 2 figures, and 2 footnotes.)