Behavioral studies have reported interactions between number processing and spatial attention, suggesting that number processing involves shifting attention along a mental continuum on which numbers are represented in ascending order. However, direct evidence for attention shifts remains scarce, the respective contribution of the horizontal and vertical axes is unclear, and little is known about the time course of attention shifts during mental manipulation of numbers. In the present study, we used an eye-tracking device with a high spatiotemporal resolution to measure gaze patterns in a task that required participants to compare number words (20 to 70) to a fixed reference (45) while looking at a blank screen (Experiment 1) or at colorful pictures (Experiment 2). Experiment 1 revealed late attention shifts evoking an epiphenomenon rather than a functional process because they occurred after the response. Experiment 2 revealed horizontal and vertical attention shifts emerging during the first stages of the comparison process. A leftward and downward ocular drift was observed while participants were listening to numbers smaller than the reference compared to numbers larger than the reference. The results showed that earlier shifts were observed when numbers were far from the reference because the decade was sufficiently discriminating to allow a fast decision. In contrast, close numbers were associated with later attention shifts because their proximity with the reference required processing the unit. We conclude that number comparison is a dynamic process that exploits visual imagery mechanisms to magnify the position of numbers on a two-dimensional space representing their magnitude.