Collaborative engagement is a quality that contributes successful learning by examining social interactions among students. In natural contexts, online collaborative learning is an evolving process that is subject to fluctuation in how students engage in social interactions. However, few studies have explored the interplay and intertwining of maintaining positive socio-emotional processes and high-level cognitive processes. Additionally, how social interactions fluctuate and transition in online collaborative learning is still unclear. In this process-oriented study, we qualitatively and quantitatively analyzed the screen-recorded collaborative learning of 15 groups of students (n = 45) and adopted the deep neural network model to automatically evaluate collaborative engagement in the online environment. The results show that neutral socio-emotional interaction is significantly associated with decreased cognitive interaction, while positive socio-emotional interaction is significantly associated with increased cognitive interaction. Furthermore, socio-emotional interactions become more positive when cognitive interactions fluctuate from deep to medium, accompanied by a relaxing group atmosphere and group members mobilizing the social climate through jokes. Cognitive interactions increase significantly when socio-emotional interactions shift from neutral to positive, mainly because positive socio-emotional interactions lead to active discussion among group members. Cognitive interactions decrease significantly when socio-emotional interactions shift from positive to neutral, mainly because the group members are less motivated and the learning goal becomes task completion instead of exploring more in-depth and comprehensive solutions. Research limitations and future research directions are also discussed concerning supporting and studying collaborative engagement in online collaborative learning.