This study investigated the relationships between sexual (verbal and non-verbal sexual harassment) and physical violence against physical education (PE) teachers and work-related stress. Participants were 175 (females 122, males 53) Finnish PE teachers aged between 27 and 62 years. The findings showed that (1) higher perceived physical violence was positively associated with higher perceived non-verbal sexual harassment and work-related stress; (2) teachers with a sport science degree perceived higher work-related stress mediated by physical violence than other teachers; (3) longer-serving PE teachers reported lower verbal sexual harassment than teachers with less teaching experience; and (4) female PE teachers reported higher work-related stress than male teachers. These results indicated that violence against PE teachers is multidimensional. To prevent work-related stress in PE teachers, school authorities could encourage teachers to report student threats as a violence prevention strategy in schools and provide appropriate support for teachers who have been victims of violence.