Purpose: We aimed to explore and compare generalist and physical education (PE) specialist (males and females) elementary teachers' self-efficacy to teach and the barriers perceived when teaching PE. Methods: Canadian elementary school teachers completed the validated online survey, Teacher Efficacy Scale in PE, with 11 additional questions examining the perceived strength of barriers related to teaching quality PE. Results: Specialist teachers' self-efficacy (n = 296) was significantly higher (p < 0.05) than that of generalist teachers (n = 818). Gender was found to predict teachers' self-efficacy, with female generalists reporting the lowest scores on the Teacher Efficacy Scale in PE. There was a statistically significant difference between the perceived strength of nine out of the 11 listed barriers, with generalist teachers reporting barriers as more inhibitory than specialists. Discussion/Conclusion: This study highlights the gap between generalists' and specialists' self-efficacy to teach and the perceived barriers when teaching PE. Efforts specifically targeted to supporting female generalists teaching PE are necessary.