Fostering productive engagement within teams has been found to improve student learning outcomes. Consequently, characterizing productive and unproductive time during teamwork sessions is a critical preliminary step to increase engagement in teamwork meetings. However, research from the cognitive sciences has mainly focused on characterizing levels of productive engagement. Thus, the theoretical contribution of this study focuses on characterizing active and passive forms of engagement, as well as negative and positive forms of engagement. In tandem, researchers have used computer-based methods to supplement quantitative and qualitative analyses to investigate teamwork engagement. Yet, these studies have been limited to information extracted primarily from one data stream. For instance, text data from discussion forums or video data from recordings. We developed an artificial intelligence (AI)-based automatic system that detects productive and unproductive engagement during live teamwork sessions. The technical contribution of this study focuses on the use of three data streams from an interactive session: audio, video, and text. We automatically analyze them and determine each team's level of engagement, such as productive engagement, unproductive engagement, disengagement, and idle. The AI-based system was validated based on hand-coded data. We used the system to characterize productive and unproductive engagement patterns in teams using deep learning methods. Results showed that there were > 91% prediction accuracy and < 7% mismatches between predictions for the three engagement detectors. Moreover, Pearson's r values between the predictions of the three detectors were > 0.844. On a scale of -1 (unproductive engagement) to 1 (productive engagement), the scores for all teams were 0.94 ± 0.04, suggesting high productive engagement. In addition, teams tended to mostly be in productive engagement before transitioning to disengagement (> 90.34% of the time) and to idle (> 93.69% of the time). Before transitioning to productive engagement, we noticed almost equal fractions of teams being in idle and disengagement modes. These results show that the system effectively detects engagement and can be a viable tool for characterizing productive and unproductive engagement patterns in teamwork sessions.