The ability to perceive approximate numerosity is present in many animal species, and emerges early in human infants. Later in life, it is moderately heritable and associated with mathematical abilities, but the etiology of the Approximate Number System (ANS) and its degree of independence from other cognitive abilities in infancy is unknown. Here, we assessed the phenotypic specificity as well as the influence of genetic and environmental factors on the ANS in a sample of 5-month-old twins (N = 514). We found a small-to-moderate but statistically significant effect of genetic factors on ANS acuity (heritability = 0.18, 95% CI: 0.02, 0.33), but only when differences in numerosity were relatively large (1:4 ratio). Non-verbal ability assessed with the Mullen Scales of Early Learning (MSEL) was found to be heritable (0.47; 95% CI: 0.34, 0.57) and the phenotypic association between ANS acuity and non-verbal ability performance was close to zero. Similarly, we found no association between ANS acuity and general attention during the task. An unexpected weak but statistically significant negative association between ANS acuity and scores on the receptive language scale of the MSEL was found. These results suggest that early ANS function may be largely independent from other aspects of non-verbal development. Further, variability in ANS in infancy seems to, to some extent, reflect genotypic differences in the population.