Musical training has long been viewed as a model for experience-dependent brain plasticity. Reports of musical training-induced brain plasticity are largely based on cross-sectional studies comparing musicians to non-musicians, which cannot address whether musical training itself is sufficient to induce these neurobiological changes or whether pre-existing neuroarchitecture before training predisposes children to succeed in music. Here, in a longitudinal investigation of children from infancy to school age (n = 25), we find brain structure in infancy that predicts subsequent music aptitude skills at school-age. Building on prior evidence implicating white matter organization of the corticospinal tract as a neural predisposition for musical training in adults, here we find that structural organization of the right corticospinal tract "in infancy" is associated with school-age tonal and rhythmic musical aptitude skills. Moreover, within the corpus callosum, an inter-hemispheric white matter pathway traditionally linked with musical training, we find that structural organization of this pathway in infancy is associated with subsequent tonal music aptitude. Our findings suggest predispositions prior to the onset of musical training from as early as infancy may serve as a scaffold upon which ongoing musical experience can build.