Undergraduate summer field programs are valuable experiences that can foster or reduce students' self-efficacy, an important factor in students' success and retention in geoscience. Growing research findings show that science field experiences can be hostile and unwelcoming to students with marginalized identities, which may negatively impact their self-efficacy in geoscience, a discipline with a dearth of students from underrepresented, marginalized identities. We conducted an interpretive qualitative study examining how summer geoscience field programs affected two undergraduate, marginalized students' self-efficacy. Adding to existing theoretical explanations of self-efficacy, we identified three types of self-efficacy impacted positively and negatively by geoscience field experiences: academic, physical, and social self-efficacy. We developed a nuanced understanding of the specific field experiences that influenced the 'ups and downs' of students' self-efficacy and, ultimately, their intent in continuing to pursue a geoscience education or career. Despite negative experiences, including gender discrimination, crude sexual jokes, and a lack of belonging, the students described their intent to persist in geoscience. Our findings can assist geoscience educators (and others in field-based sciences) to consider experiences that support and hinder marginalized students' self-efficacy. Also, our findings can guide efforts to improve geoscience field programs to create more inclusive environments.