Listeners are known to track statistical regularities in speech. Yet, which temporal cues are encoded is unclear. This study tested effects of talker-specific habitual speech rate and talker-independent average speech rate (heard over a longer period of time) on the perception of the temporal Dutch vowel contrast /?/-/a:/. First, Experiment 1 replicated that slow local (surrounding) speech contexts induce fewer long /a:/ responses than faster contexts. Experiment 2 tested effects of long-term habitual speech rate. A high-rate group listened to ambiguous vowels embedded in "neutral" speech from Talker A, intermixed with fast speech from Talker B. A low-rate group listened to the same neutral speech from Talker A, and/but to Talker B speaking at a slow rate. Between-groups comparison of the neutral trials showed that the high-rate group demonstrated a lower proportion of /a:/ responses, indicating that Talker A's habitual speech rate sounded slower when B was faster. In Experiment 3, both talkers produced speech at both rates, removing the different habitual speech rates of Talkers A and B, while maintaining the average rates differing between groups. In Experiment 3, no global rate effect was observed. Taken together, the present experiments show that a talker's habitual rate is encoded relative to the habitual rate of another talker, carrying implications for episodic and constraint-based models of speech perception.