The National Assessment Program--Literacy and Numeracy (NAPLAN) assessments are designed to assess literacy and numeracy of all Australian school children in years 3, 5, 7 and 9, and to act as diagnostics as to whether children are meeting intended educational outcomes. Tests began in May 2008, and have been run annually since then. Results of the 2008 tests indicated that Indigenous children in remote communities had the lowest test scores, and results were used to make a policy decision that effectively scrapped bilingual education in the Northern Territory. In this paper, we evaluate the literacy component of the NAPLAN test for Year 3, and the language samples for each year level. Literacy components assess reading, writing and language conventions (grammar,spelling and punctuation), and we focus on the reading and language conventions components. We argue that the NAPLAN tests need to be very carefully monitored for appropriateness for the assessment of children living in remote Indigenous communities. This is because tests are standardised on groups of English language speaking children. The content of some sample tests relies on cultural knowledge which Indigenous children cannot be expected to have. Spelling tests need to be monitored to ensure that they are testing spelling rather than grammatical knowledge. Finally, it is difficult to create language convention tests which are truly diagnostic because of the mixed test population of native English speakers, ESL learners and EFL learners in remote Indigenous communities. (Contains 17 tables and 3 endnotes.)