In this study, we explore the influence of student engagement on middle school students' environmental literacy outcomes at a residential environmental education (EE) program. We observed 80 lessons and 17 educators, observing six measures related to engagement. We also administered immediate post-experience surveys measuring students' self-reported levels of engagement and environmental literacy outcomes. Only self-reported student engagement was significantly associated with more positive outcomes. As such, observations of engaged or enthusiastic EE participants may not necessarily be indicative of achieving desired learning outcomes. The study reveals the importance of systematic evaluation over simplistic observations for gauging participant learning.