One core question in studies of language processing is the extent to which interlocutors engage in real-time communicative perspective-taking. Current evidence suggests that both children and young adult listeners are able to draw on common ground (shared knowledge) to guide referential interpretation. However, less is known about older listeners, who are often described as experiencing age-related cognitive declines that could affect their capacity to integrate perspective cues online. In the present study, we examined the extent to which younger and older listeners used common ground to guide the interpretation of temporarily ambiguous descriptions. Participants followed instructions from a Director to click on displayed objects. The target object (e.g., hat with blue feathers) was accompanied by a competitor (e.g., hat with pink feathers) or a control object (e.g., stapler). We manipulated whether the competitor/control was mutually visible (common ground) or not (privileged ground). The results revealed that, although listeners used perspective information to differentiate the target from the competitor in the common ground condition, this pattern was notably weaker in older adults. Whereas measures of executive function showed significant group differences in inhibitory control and working memory, no differences were found in theory of mind. Thus, age-related changes in communicative perspective-taking are not likely due to general declines in mentalizing ability. Furthermore, strict screening criteria for vision and hearing ability allowed us to rule out explanations involving age-related sensory decline. Together, the results advance our understanding of how younger and older adults integrate common ground during real-time referential processing.