Learning or performance gains are often measured in improvements of test scores over time. However, learners may also answer questions incorrectly which they appeared to master previously, making claims about overall learning gains problematic. Using data from 1,211 grade 6 learners in the Umgungundlovu district in KwaZulu-Natal, we interrogated the consistency of the improvement of their test scores after a year in grade 6. The learners had completed the same 40 question multiple choice test at the onset and the end of grade 6, with no interventions only normal schooling in-between. The content of the test was grade 5 and 6 mathematical content. We know from previous research that the learners were not randomly guessing, yet we found that on average, the learners only got 6 out of 40 questions correct on both test 1 and test 2, did worse on 5 questions and better on 6. Further, almost half the learners both improved on 5 or more questions and declined on 5 or more questions. On 35 questions, more than half the learners changed their answers between the two tests. In other words, their performance is very 'unstable'. This indicates that learners are not consistent in their thinking, possibly reflecting guessing, albeit not random. Furthermore, the findings point to a methodological issue of making claims about average performance gains, as these may disguise very unstable learner performance. Such claims should therefore be interrogated before they are applied in making statements about links between learning and teaching or other background factors.