Bilingual infants acquire languages in a variety of language environments. Some caregivers follow a one-person-one-language approach in an attempt to not "confuse" their child. However, the central assumption that infants can keep track of what language a person speaks has not been tested. In two studies, we tested whether bilingual and monolingual 5-, 12- and 18-month-olds spontaneously form language-person associations. In both studies, infants were familiarized with a man and a woman, each speaking a different language, and tested on trials where they either spoke the same language or switched to a different language. In Study 1, infants only heard the speaker, and in Study 2, infants saw and heard the speaker. Bilinguals and monolinguals did not look longer for Switch compared to Same trials; there was no evidence in this task that infants form person-language associations spontaneously. Thus, our results did not support a central assumption of the one-person-one-language approach, although we cannot rule out that infants do form this association in more naturalistic contexts.