Previous studies on spoken word production have shown that native English speakers used phoneme-sized units (e.g., a word-initial phoneme, C) to produce English words, and native Mandarin Chinese speakers employed syllable-sized units (e.g., a word-initial consonant and vowel, CV) as phonological encoding units in Chinese. With spoken word production, although spoken word recognition is a fundamental language skill of human beings, it is unknown whether a Chinese-English bilingual listener can adjust the phonological unit sizes depending on the language used. Using four mouse-tracking spoken word recognition experiments, participants listened to spoken words (either presented in English or Chinese) while viewing a display of two written words. In Experiment 1, Chinese-English bilinguals experienced a larger phonological competition in the CV condition than the control condition, indicating that they recruited syllables when listening to Chinese monosyllabic words. Experiment 2 extended the results of Experiment 1 with a set of Chinese disyllabic words and further revealed the temporal distribution of syllable overlap of a spoken word constrains phonological competition. In Experiment 3, Chinese-English bilinguals exhibited a greater phonological competition in both CV and C conditions that mirrored those reported with English monolinguals in Experiment 4. Our results across experiments demonstrated that Chinese-English bilinguals employed both phonemes and syllables to recognize English spoken words but exclusively relied on syllables to recognize Chinese words, suggesting that there is flexibility in selecting phonological units when recognizing spoken words in the two languages. Implications for models of monolingual and bilingual spoken recognition are also discussed.