Purpose: Congenital amusia is an inborn neurogenetic disorder of fine-grained pitch processing. This study attempted to pinpoint the impairment mechanism of speech processing in tonal language speakers with amusia. We designed a series of perception tasks aiming at selectively probing low-level pitch processing and relatively high-level phonological processing of lexical tones, with an aim to illuminate the deficiency mechanism underlying tone perception in amusia. Method: Sixteen Cantonese-speaking amusics and 16 matched controls were tested on the effects of acoustic (talker/syllable) variations on the identification and discrimination of Cantonese tones in two conditions. In the low-variation condition, tones were always associated with the same talker or syllable; in the high-variation condition, tones were associated with either different talkers (with the syllable controlled) or different syllables (with the talker controlled). Results: Largely similar results were obtained in talker and syllable variation conditions. Amusics exhibited overall poorer performance than controls in tone identification. Although amusics also demonstrated poorer performance in tone discrimination, the group difference was more obvious in low-variation conditions, where more acoustic constancy was provided. Besides, controls exhibited a greater increase in discrimination sensitivity from high- to low-variation conditions, implying a stronger benefit of acoustic constancy. Conclusions: The findings suggested that amusics' lexical tone perception abilities, in terms of both low-level pitch processing and high-level phonological processing, as measured in low- and high-variation conditions, are impaired. Importantly, amusics were more impaired in taking advantage of low acoustic variation contexts and thus less efficiently sharpened their perception of tones when perceptual anchors in talker/syllable were provided, suggesting a possible "anchoring deficit" in congenital amusia.