Subjects such as Health and Physical Education (HPE) can be marginalised in schools because they are construed as less academically rigorous and less important to the primary mission of education. Teachers of all subjects face challenges, yet, teaching a marginalised subject can result in additional challenges for HPE teachers. Previous research has noted these challenges; however, less is known about how these challenges vary according to student age and teacher experience. This study used quantitative survey methods to ascertain which challenges are the most difficult for Australian HPE teachers, and whether this difficulty varies according to their teaching experience and the ages of their students. Findings indicated that the year level of students taught by HPE teachers was significantly associated with student engagement and isolation within their schools. Years of teaching experience was significantly associated with challenges in teaching students with special needs, with more experienced teachers rating this area as less of a challenge than less experienced teachers.