Does tracking another agent's visual perspective depend on having a goal--albeit a remote one--to do so? In 5 experiments using indirect measures of visual perspective taking with a cartoon avatar, we examined whether and how adult perceivers' processing goals shape the incidental tracking of "what" objects the avatar sees (Level-1 perspective taking) and "how" the avatar sees those objects (Level-2 perspective taking). Process dissociation analyses, which aim to isolate calculation of the avatar's perspective as the process of focal interest, revealed that both Level-1 and Level-2 perspective calculation were consistently weaker when the avatar's perspective was less relevant for participants' own processing goals. This pattern of goal-dependent perspective tracking was also evident in behavioral analyses of interference from the avatar's differing perspective when reporting one's own perspective (i.e., altercentric interference). These results suggest that, although Level-1 and Level-2 visual perspective calculation may operate unintentionally, both also appear to depend on perceivers' processing goals. More generally, these findings advance understanding of processes underlying visual perspective taking and the conditional automaticity with which those processes operate.