The young universal listener is an established concept in psycholinguistics. However, it is unclear what abilities universal perception entails and at what age it exists. This article aims to motivate rethinking about what it means to be a universal listener. Early and recent studies on infant speech acquisition are reviewed, considered in the light of cross-language variation and adult performance, and finally linked to the current understanding of fetal hearing and learning. It turns out that language-universal perception is best described as an auditory-based perception rather than an ability to perceptually categorize the sounds of any possible language. Interestingly, at birth infants might no longer listen in a language-universal mode because learning from the ambient speech signal begins at least several weeks before birth. Future studies need to answer the remaining questions concerning the point in perinatal development at which speech perception begins to take on language-specific traits and for which sounds.