Mathematical giftedness (MG) is an intriguing phenomenon, the nature of which has yet to be sufficiently explored. This study goes a step further in understanding how MG is related to expertise in mathematics (EM) and general giftedness (G). Cognitive testing was conducted among 197 high school students with different levels of G and of EM. Based on our previous studies, we perceive MG as a combination of G and EM. Exploratory factor analysis of test results revealed five main cognitive factors: visual-serial processing (VSP); arithmetic abilities (AA); pattern recognition (PR); auditory working memory (AWM); visual-spatial working memory (VSWM); and Structural equation modeling (SEM) based on the factor analysis revealed clear differences in the role of cognitive abilities as predictors of EM, G, and MG. The study demonstrates that visual components are especially important for the development of EM and that G students are less dependent on their visual cognitive processing. Based on the study results, we argue that EM, G, and MG, which are often considered equivalent characteristics, are interrelated but different in nature. The paper presents a research-based justification that not any gifted is an expert in mathematics and not any expert in mathematics is gifted.