Efficient neural encoding of sound plays a critical role in speech and language, and when impaired, may have reverberating effects on communication skills. This study investigated disruptions to neural processing of temporal and spectral properties of speech in individuals with ASD and their parents and found evidence of inefficient temporal encoding of speech sounds in both groups. The ASD group further demonstrated less robust neural representation of spectral properties of speech sounds. Associations between neural processing of speech sounds and language-related abilities were evident in both groups. Parent-child associations were also detected in neural pitch processing. Together, results suggest that atypical neural processing of speech sounds is a heritable ingredient contributing to the ASD language phenotype.