We examined the validity of the comprehension component of the Gray Oral Reading Test (GORT; Wiederholt & Bryant, 1992, 2001) by assessing whether reading really is required to answer its questions. The extent to which GORT questions are passage independent was assessed by having participants answer them without reading the passages. Most questions had passageless accuracies above chance. Furthermore, the best predictor of how well a child given normal GORT administration answered a question was not how well the child read the passage but rather how well the question could be answered without reading. Analyses comparing passage-dependent and passage-independent items showed: (a) passage-independent items are not sensitive to reading disability, and (b) passage-independent items do not correlate with performance on other comprehension tests. We conclude that the GORT Comprehension Score lacks both content validity and concurrent validity and that the field of comprehension assessment needs to be more concerned about the passage independence of items.