This study proposes a formal multi-step methodology for qualitative assessment of topic modeling results in the context of online learner motivation to purchase Statements of Participation (SoP). We developed Latent Dirichlet Allocation (LDA) based topic models on open-ended responses of three post-course survey questions from 280 open courses offered on the FutureLearn learning platform. For qualitative assessment, we first determined the theme of the topic based on the words that constituted the topic and responses that were most strongly associated with the topic. Then, we verified the theme by comparing the topics assigned by LDA model on a test set with manual annotation. We also performed sentiment analysis to check for alignment with human judgment. Learner motivations in each theme were interpreted with the Expectancy-Value-Cost framework. Our analyses indicated that, primarily, learners were motivated to purchase the SoP based on perceptions of the utility value and financial cost of the certificate. We found that human judgment agreed with the topic model more frequently when LDA topic weights were larger. [For the full proceedings, see ED593090.]