International and national testing of numerical and language skills has become a regular part of educational systems in many countries. In Australia, the "National Assessment Program--Literacy and Numeracy" (NAPLAN) has been used since 2008 to carry out regular tests of literacy and numeracy amongst school students nationally. Since the numeracy components of this program are based on work carried out in school mathematics classes, it seems reasonable that tests represent an evaluation of mathematical ability, albeit at the simpler and introductory levels. However, there has been little investigation of students' ideas about numeracy, and the role that their attitudes towards numeracy may have on their results on the numeracy components of NAPLAN tests. This study carries out an empirical investigation of ideas about and attitudes towards numeracy, and their relation to NAPLAN scores for a sample of 735 lower secondary students from two schools in New South Wales, Australia. Attitudes are measured using a modified form of the "Students' Attitudes Towards Statistics," (SATS-36) test (Schau in Survey of attitudes towards statistics, http://www.evaluationandstatistics.com, 2003), and conceptions of numeracy are obtained from phenomenographic analysis of an open-ended response item. The overall conclusion, that students' understanding of the concept of numeracy and their attitudes towards numeracy are related to their performance in numeracy tests, represents a potentially important result both for students and their teachers.