Background: Previous research showed a significant association between mathematics and working memory (WM). However, evidence regarding the different effects of verbal and visuospatial WM on mathematical abilities was very limited. Aims: The current research aims to explore the relationship between verbal and visuospatial WM with mathematical abilities, and how this relationship is moderated by age and math domains. We also wonder whether the results would change when we use several tests for each component and use a latent variable approach for more reliable measurement. Sample: 131 first graders and 144 fifth graders from a primary school in Hangzhou, Zhejiang province, China, participated our research. Methods: All participants completed three verbal and three visuospatial WM tasks, mathematics tests, and fluid intelligence test. Hierarchical multiple regression analysis was used to examine the pattern of relations among these constructs. Results: The results of hierarchical multiple regression analysis showed that verbal WM significantly predicted mathematics achievement for fifth graders, while failed for first graders. However, visuospatial WM played a substantive role in both graders' mathematical performance. The different role of the two WM components also depend on various fields of mathematics. Conclusion: Our results indicated the distinct influence of verbal and visuospatial WM on primary school students' mathematical abilities and highlighted the developmental and domain-specific effects of WM on mathematics.