Previous studies have found that good physical exercise can promote academic performance, but the underlying mechanism behind this lacks large-scale empirical data. Based on this, we used the 2020 Jiangsu Province academic quality monitoring data to construct an OLS regression. Then, the non-cognitive abilities variable was built with the NEO-FFI Theory, and the mediation effect diagram was drawn through the Amos22.0 software. The study found that: (i) Physical exercise brought academic improvement to elementary and middle school students. (ii) The influence mechanism of physical exercise was different between elementary school and middle school. Students at the elementary school were directly and indirectly affected. (iii) In non-cognitive specific dimensions, openness played a significant and great mediating role.