As an important branch of mathematics, geometry plays a very significant role in developing students' thinking, but many students need to improve their geometric thinking abilities, especially in understanding graphics and forms. Therefore, it is worthwhile to explore techniques and methods for developing students' geometric thinking, and Augmented Reality (AR) technology can visualize geometric objects to enhance mathematic learning. This study designed an interactive mathematical picture book by utilizing AR technology, and used a quasi-experimental setting to compare the effects of three different reading materials, including AR mathematical picture books, mathematical picture books and mathematical texts. Eighty-three fourth-grade students at an elementary school in China were divided into three groups to participate in the experiment. The effects on primary school students were measured in terms of geometric thinking level, cognitive load and flow experience. The quantitative findings indicated that after reading the AR mathematical picture book, students' geometric thinking levels 0-3 were significantly improved (p < 0.05). Moreover, compared with the other two groups, the students who finished reading the AR mathematical picture book scored significantly higher in geometric thinking and flow experience and significantly lower in cognitive load. Analysis of the semi-structured interviews showed that students enjoyed using AR mathematical picture books and felt that it resulted in a better flow experience and lower cognitive load. As a result, this research indicates that AR mathematical picture book has a significant role in increasing students' geometrical thinking, and provide a successful teaching case for the application of AR/VR in the classroom.