Visual object tracking is an essential technique for constructing intelligent livestock management systems. Behavior patterns estimated from the trajectories of animals provide substantial useful information related to estrus cycle, disease prognosis and so on. However, similar colors and shapes between animals often lead to the failure of tracking multiple objects, and the background clutter of the breeding space further makes the problem intractable. In this paper, we propose a novel method for tracking animals using a single thermal sensor. The key idea of the proposed method is to represent the foreground (i.e., animals) easily obtained by a simple thresholding in a thermal frame as a topographic surface, which is very helpful for finding the boundary of each object even in cases with overlapping. Based on the segmentation results derived from morphological operations on the topographic surface, the center positions of all the animals are consistently updated with an efficient refinement scheme that is robust to the abrupt motions of animals. Experimental results using various thermal video sequences demonstrate the efficiency and robustness of our method for tracking animals in a breeding space compared to previous approaches proposed in the literature.