Previous research shows a correlation between individual differences in people's school math abilities and the accuracy with which they rapidly and nonverbally approximate how many items are in a scene. This finding is surprising because the Approximate Number System (ANS) underlying numerical estimation is shared with infants and with non-human animals who never acquire formal mathematics. However, it remains unclear whether the link between individual differences in math ability and the ANS depends on formal mathematics instruction. Earlier studies demonstrating this link tested participants only after they had received many years of mathematics education, or assessed participants' ANS acuity using tasks that required additional symbolic or arithmetic processing similar to that required in standardized math tests. To ask whether the ANS and math ability are linked early in life, we measured the ANS acuity of 200 3- to 5-year-old children using a task that did not also require symbol use or arithmetic calculation. We also measured children's math ability and vocabulary size prior to the onset of formal math instruction. We found that children's ANS acuity correlated with their math ability, even when age and verbal skills were controlled for. These findings provide evidence for a relationship between the primitive sense of number and math ability starting early in life.