Background: South African primary school learners have participated in several national and international literacy (reading and writing) studies that measure learners' achievement in different grades and at different intervals. Numerous scholars have analysed the results of these assessments. We extended their analyses by investigating the grade coverage and the aspects of literacy that were included in these assessments, as well as whether the learners' home language impacts their results. Aim: The authors aim to determine how reliable the results of these assessments are in improving and informing policies relating to literacy teaching in primary schools and to provide recommendations to improve the administration of literacy assessments. Method: Literature on various national and international literacy studies that were conducted in South African primary schools from 2000 to 2016 was identified and analysed according to grade, province, languages in which the assessments were conducted, aspects of literacy that were included in the assessments and the accuracy of the results. Results: The analysis provides evidence that suggests that most literacy assessments target learners in the Intermediate Phase (Grades 4-6) and are not available in all 11 South African official languages. Presently, there are no large-scale literacy assessments for Foundation Phase (Grades 1-3) learners. Moreover, the results of these assessments do not provide us with reliable information about literacy levels in the country because there are vast discrepancies in assessment scores. Conclusion: This article highlights the importance of obtaining reliable information in determining literacy levels in the country and in informing decisions regarding literacy-related policies.