Running Records provide a standardized method for recording and assessing students' oral reading behaviors and are excellent formative assessment tools to guide instructional decision-making. This study expands on prior Running Record reliability work by evaluating the extent to which external raters and teachers consistently assessed students' accuracy and self-correction ratios in the context of a naturalistic lesson setting. This study was conducted using recorded lessons from 19 students, taught by 11 teachers in ten schools, who in separate lessons read two books at the same text level. Generalizability theory framework was applied to identify sources of variability in accuracy and self-correction scores and subsequently evaluate necessary conditions for reliable measurement using Running Records. Results indicated that raters were highly reliable. However, due to the impact of reading occasion, approximately eight to ten reads are needed for reliable assessment of students' accuracy and self-correction via Running Records.