Recognizing learners' motivational problems in online remote learning, this study examined the patterns of learner's academic emotions and facial expressions detected in asynchronous video-based learning. The research questions of this are: (1) What are the patterns between facial expressions and negative emotions (i.e., boredom, confusion, and frustration) that learners experienced during online video-based learning? (2) What are the learner's overall perceptions about their emotions during online video-based learning? This study was conducted with 26 Korean adult learners who took an online video lecture via an asynchronous self-directed mode. Their facial expressions during online learning were recorded and analyzed for detecting affective states. Two judges trained in Ekman's Facial Action Coding System (FACS) analyzed 210 scenes segmented from the recorded video data using a retrospective affect judgment procedure. The analysis identified 101 incidences of boredom, 82 confusions, and 62 frustrations. Moreover, using Spearman's rank correlation coefficients and hierarchical clustering, we identified some patterns of significant relationships between facial action units and affective states. The interview data reveal that the potential reasons for negative emotions include lack of concentration, uninteresting content, and one-way content delivery method. Given the massive transition to online video-based learning during and after COVID-19, the results of this study can provide implications concerning how to detect and intervene in potential issues associated with learner's motivation and affective states.