The introduction of the Annual National Assessment (ANA) programme by the South African education department was met with much resistance from unions with some stating that it is not matched well to the curriculum. In this small-scale study, I compare the differences in outcomes between the internal summative assessment and the ANA for Grade 9 learners from five schools. The results indicate that the learners' results from the final exam were statistically significantly higher than that from the ANA for all five schools. This suggests that the ANA was considerably more difficult than the schools' assessments. However, the learners' scores in each of the two assessments were strongly correlated, suggesting that both tests assessed similar skills. Interviews with the teachers suggested that some of the reasons for the lower results in the ANA were that the test was too long, some instructions were not clear, the timing in September was not ideal and there were too many higher level questions. It is hoped that the results from this study could be used by the education department to improve the functioning of the assessment programme.