Listeners can access information about a speaker such as age, gender identity, socioeconomic status, and their linguistic background upon hearing their speech. However, it is still not clear if listeners use these factors to assess speakers' speech. Here, an audio-visual (matched-guise) test is used to measure whether listeners' accentedness judgments can be modulated depending on the type of face that they see. American and Indian English were used as different English varieties and presented with either a White female face or a South Asian female face. Results show that listeners' accentedness judgments increased for Indian English compared to American English. Importantly, the increase in accentedness judgments was also observed when both American English and Indian English were presented with a South Asian face compared to a White face. These findings suggest that linguistic evaluations are modulated by non-linguistic factors and that speech perception is socially gated.