Infants struggle to understand familiar words spoken in unfamiliar accents. Here, we examine whether accent exposure facilitates accent-specific adaptation. Two types of pre-exposure were examined: video-based (i.e., listening to pre-recorded stories; Experiment 1) and live interaction (reading books with an experimenter; Experiments 2 and 3). After video-based exposure, Canadian English-learning 15- to 18-month-olds failed to recognize familiar words spoken in an unfamiliar accent. However, after face-to-face interaction with a Mandarin-accented talker, infants showed enhanced recognition for words produced in Mandarin English compared to Australian English. Infants with live exposure to an Australian talker were not similarly facilitated, perhaps due to the lower vocabulary scores of the infants assigned to the Australian exposure condition. Thus, live exposure can facilitate accent adaptation, but this ability is fragile in young infants and is likely influenced by vocabulary size and the specific mapping between the speaker and the listener's phonological system.