The aim of this study was to investigate the quality of the Annual National Assessment (ANA) Grade 6 Mathematics assessment instrument, administered in the year 2012 guided by the main question: to what extent does the 2012 Grade 6 Mathematics assessment instrument provide meaningful and consistent information for making appropriate interpretations? The responses to 29 questions, comprising 56 items, from 546 learners' scripts were coded individually, recaptured and analysed. The analysis shows that the Annual National Assessment was not well targeted, that is, the mean of the item and the mean of the person measures were not approximately equivalent. Specifically, in relation to this cohort, and taking into account the many factors that influence performance, we recommend that for reliable inferences to be made from test results, the quality of the assessment instrument should first be established.