Students' perceptions of their mathematics ability vary by gender and seem to influence science, technology, engineering, and mathematics (STEM) degree choice. Related, students' perceptions during academic difficulty are increasingly studied in educational psychology, suggesting a link between such perceptions and task persistence. Despite interest in examining the gender disparities in STEM, these concepts have not been considered in tandem. In this manuscript, we investigate how "perceived ability under challenge"--in particular in mathematics domains--influences entry into the most sex-segregated and mathematics-intensive undergraduate degrees: physics, engineering, mathematics, and computer science (PEMC). Using nationally representative Education Longitudinal Study of 2002 (ELS) data, we estimate the influence of perceived ability under challenging conditions on advanced high school science course taking, selection of an intended STEM major, and specific major type 2 years after high school. Demonstrating the importance of specificity when discussing how gender influences STEM career pathways, the intersecting effects of gender and perceived ability under mathematics challenge were distinct for each scientific major category. Perceived ability under challenge in secondary school varied by gender, and was highly predictive of selecting PEMC and health sciences majors. Notably, women's 12th grade perceptions of their ability under mathematics challenge increased their probability of selecting PEMC majors over and above biology. In addition, gender moderated the effect of growth mindset on students' selection of health science majors. Perceptions of ability under challenge in general and verbal domains also influenced retention in and declaration of certain STEM majors. The implications of these results are discussed, with particular attention to access to advanced scientific coursework in high school and interventions aimed at enhancing young women's perceptions of their ability, in particular in response to the potentially inhibiting influence of stereotype threat on their pathways to scientific degrees.