The purpose of this study was to compare traditional oral reading fluency (ORF) measures to a computerized oral reading evaluation (CORE) system that uses speech recognition software. We applied a mixed model approach with two within-subject variables to test the mean WCPM score differences and the error rates between: passage length (25, 50, 85, and 250 words), and scoring method (real-time, recorded audio, and automated speech recognition [ASR]). We also used weighted Cohen's kappa to analyze the word score agreement across scoring methods. We found that shorter passages had greater WCPM means than traditional ORF passages, and that real-time (traditional) scores had greater means than both audio recording and ASR scores; the latter two were similar across passage length and grade.