In the UK, one consequence of neoliberalism has been the development of test cultures in schools and standardised assessment strategies used to judge all pupils against within and across curriculum subjects. Few studies to date have explored the influence of this on assessing the learning of pupils with special educational needs and disabilities (SEND), and none have centred physical education (PE). This study used the concept of ableism and semi-structured interviews to explore mainstream secondary school PE teachers' views and experiences of assessing the learning of pupils with SEND. Based on the findings, we discuss the importance of schools disrupting hegemonic, ableist modes of thinking that cast pupils with SEND as being of inferior ability when compared with their peers and thus being disadvantaged by standardised, normative assessment practices. Specifically, we identify a need for senior leaders and teachers in schools to recognise the needs and capabilities of pupils with SEND, through more holistic assessment approaches that focus on social, affective, cognitive and physical learning and development. We end by discussing the significance of initial teacher education and teacher networks to support this endeavour and advocating for the amplification of the voices of pupils with SEND, given that they have expert knowledge about the perceived inclusivity of assessment in PE because they can draw upon their lived and embodied experiences.