This study examined the reliability and validity of scores on a fluency-based measure of reading comprehension. The Dynamic Indicators of Basic Early Literacy Skills (DIBELS; 6th ed. revised) Retell Fluency (RTF), Oral Reading Fluency (DORF), and Woodcock Johnson III NU Tests of Achievement (WJ-III NU ACH) Reading Comprehension measures were administered to fourth-grade students. Results indicated a large difference between real time and recorded retell fluency scores for each passage. In addition, students' retell fluency scores had a low correlation with their reading comprehension scores. In light of these findings, practitioners may want to exercise caution in using fluency-based story-retell scores as a measure of reading comprehension. (Contains 3 tables and 1 footnote.)