In this opinion piece, I discuss a range of issues associated with the inclusion of gender diverse students' responses in the analyses of large-scale mathematics education data sets. Definitions of the terms gender and sex over time are examined, as well as the associated gender and sex categories that have and can be used. Of particular interest are the challenges confronting researchers of gender issues in mathematics education who are sympathetic to gender inclusivity and want to conduct statistical analyses of large-scale data sets. In light of the statistical limitations described, some suggestions are proposed for overcoming them.