The phonological deficit theory of dyslexia assumes that degraded speech sound representations might hamper the acquisition of stable letter-speech sound associations necessary for learning to read. However, there is only scarce and mainly indirect evidence for this assumed letter-speech sound association problem. The present study aimed at clarifying the nature and the role of letter-speech sound association problems in dyslexia by analysing event-related potentials (ERP) of 11-year-old dyslexic children to speech sounds in isolation or combined with letters, which were presented either simultaneously with or 200 ms before the speech sounds. Recent studies with normal readers revealed that letters systematically modulated speech sound processing in an early (mismatch negativity or MMN) and late (Late Discriminatory Negativity or LDN) time-window. The amplitude of the MMN and LDN to speech sounds was enhanced when speech sounds were presented with letters. The dyslexic readers in the present study, however, did not exhibit any early influences of letters on speech sounds even after 4 years of reading instruction, indicating no automatic integration of letters and speech sounds. Interestingly, they revealed a systematic late effect of letters on speech sound processing, probably reflecting the mere association of letters and speech sounds. This pattern is strongly divergent from that observed in age-matched normal readers, who showed both early and late effects, but reminiscent of that observed in beginner normal readers in a previous study (Froyen, Bonte, van Atteveldt & Blomert, 2009). The finding that the quality of letter-speech sound processing is directly related to reading fluency urges further research into the role of audiovisual integration in the development of reading failure in dyslexia.