Purpose: This study investigated the categorical perception of Mandarin tones and consonant aspiration contrasts in babble noise among adults and adolescents aged 12-14 years, and explored the association between working memory and categorical perception. Method: Twenty-four adults and 20 adolescents with Mandarin as their native language were recruited. Their performances of phonemic identification and discrimination in babble noise and quiet conditions, digit span tasks, and nonword repetition were assessed. Results: Results indicated that, firstly, in the noise condition, both adults and adolescents showed wider boundary widths and lower between-category accuracies when perceiving aspiration of consonants than in the quiet condition, and the categorical boundary of tone perception in adolescents showed a transitional tendency toward Tone 1. Secondly, discrimination of consonant aspiration in adolescents needed to be further developed. Lastly, the accuracy of nonword repetition in adolescents was lower than that in adults, and adults with better auditory verbal working memory had better performance on tone perception. Conclusions: Our results provided evidence that tone perception is acquired easier than consonant aspiration perception, and tone perception is more robust and less susceptible to noise interference. Categorical perception performance relates to the capacity and utilization of auditory verbal working memory in some ways.