Computer-based curriculum-based measurement (CBM) is a relatively common practice, but surprisingly few studies have examined the reliability of computer-based CBM. This study sought to examine the reliability of CBM administered via paper/pencil versus the computer. Twenty-one of 25 students in two third-grade classes (N = 21) participated in two generalizability studies. The primary facet of interest, format of assessment, was examined for two measures, a fact operation (fact family measure) and a more complex operation (multidigit addition). Researchers administered four alternate forms of each measure under both computer and paper/pencil conditions. Results indicated ideal reliability of measurement within all conditions and results were consistent across assessment formats for the fact operation. However, assessment format explained a large proportion of variance for the more complex skill (multidigit addition). Results indicate a need to consider the reliability of transitioning assessment from paper/pencil to computer-based response on a measure-by-measure basis.