Although intelligibility is a core concept in second language (L2) speech assessment and teaching research, the vast majority of previous work relies on audio-only stimuli. The current study set out to examine how linguistic and visual information jointly interact to determine the degree of speech intelligibility. Both first language (L1) and L2 English listeners were presented with stimuli that varied along 3 factors (vowel error, visual speech, and iconic gesture) and completed an orthographic transcription task. Results revealed that iconic gesture significantly increased all the listeners' intelligibility scores when speech contained vowel errors. When speech did not contain errors, gesture increased intelligibility for L2 listeners but not L1 listeners. Visual speech had no significant effect on intelligibility in either listener group. Vowel error reduced intelligibility by approximately 20-30% for both L1 and L2 listeners. Findings suggest that visual modalities, especially gestures, have the potential to significantly affect the intelligibility of speech containing phonological errors.