Given the inadequacy of assessed outcomes (e.g., final exam) and the importance of evaluating the learning process in STEM education, we use deep learning to develop the STEM learning behavior analysis system (SLBAS) to assess the behavior of learners in STEM education. We map learner behavior to the ICAP (interactive, constructive, active, passive) framework, helping instructors to better understand the learning process of learners. The results show that SLBAS exhibits high accuracy. Moreover, Cohen's kappa coefficient between expert coding and SLBAS is high enough to support replacing expert coding in the observation method with SLBAS to recognize the learning process of learners during STEM activities. Finally, statistical analysis establishes a correlation between the learning process and learning effectiveness. The results of this study are in line with most previous studies, demonstrating that STEM education differs from traditional teacher-centered courses in that it helps learners to improve the process of knowledge construction with practice and hands-on opportunities rather than simply receiving knowledge passively.