Barriers to women's education and employment in Europe and the United States in the 19th century made it unlikely that any women would be among the few physicists whose ideas are taught in high school and college courses. This paper explores the social settings in which three influential physicists worked--James Clerk Maxwell, Robert Millikan, and Albert Einstein--to better understand the limited opportunities available to women. By acknowledging and explaining why there were not more women among these founding physicists, instructors may help students understand the barriers that still exist and feel more empowered to overcome them and pursue physics as a career.