The purpose of this study was to compare students' results in mathematics from a large-scale standardized assessment, the National Assessment Program: Literacy and Numeracy (NAPLAN), with a set of teacher-developed, school-based assessments. A case study of an all-boys secondary school in Sydney, New South Wales, Australia, was conducted over three years with a total 1,456 student-participants. We found strong positive correlations existed between the NAPLAN data and certain school-based assessment data, such as monthly tests, but such results were not consistent across all classes. We conclude that NAPLAN data when considered in isolation, might be of limited benefit to teachers and students for diagnostic purposes. We therefore offer practical suggestions as to how student performance data generated from a large-scale assessment like NAPLAN might be best utilized and interpreted for formative assessment purposes in the school to optimally benefit individual students' learning.