Math anxiety (MA) is an important affective factor that contributes to individuals' math proficiency. While self-reports are commonly used to measure MA, a number of limitations are inherently connected to this measuring method. Physiological responses are considered a promising alternative approach, but research is scarce and the empirical evidence is scattered. Therefore, this paper aimed to (1) investigate whether different types of tasks (i.e., difficulty and topic) result in differences regarding self-reported anxiety and physiological measures, and (2) analyse whether physiological measures can account for differences in self-reported MA. We manipulated the difficulty level of a math and non-math task, so this study had a two-by-two experimental within-subject design. The participants were 44 undergraduate students. In terms of the first research aim, results revealed that the difficult math task elicited more self-reported anxiety compared to the easy math task and the difficult non-math task. However, these differences are barely detected by physiological measures. Regarding the second research aim, results showed that phasic galvanic skin responses and heart coherence ratio significantly predicted the self-reported MA. Our findings point to a possible contribution of using physiological measures to understand the construct of MA, meanwhile warning for a too optimistic use of this measurement method.