Culturally relevant pedagogy (CRP) seeks to improve equity in instruction and leverage students' experiences by promoting academic success, cultural competence, and sociopolitical consciousness. We examine instructors' perceptions of student identity to understand the ways undergraduate mathematics instructors are enacting or experiencing barriers to enacting CRP. Interviews with ten mathematics faculty at Hispanic-serving institutions identified two potential barriers to enacting CRP: first, instructors' hesitance to communicate about student identity, especially with respect to race and gender; and second, instructors holding epistemologies that mathematics is culture-free. Despite these barriers, almost all interviewees implemented the academic success tenet of CRP. These barriers may prevent instruction around cultural competence and sociopolitical consciousness, which are the two tenets that most capitalize on students' informal knowledge, identities, and cultural experiences. Changing discourse by taking more risks in conversation and inviting a more diverse range of people to the undergraduate mathematics community are potential ways to address these barriers.