The use of machine translation (MT) in the academic context has increased in recent years. Hence, language teachers have found it difficult to ignore MT, which has led to some concerns. Among the concerns, its accuracy has become a major factor that shapes language teachers' pedagogical decision to use MT in their language classrooms. Despite the urgency of the issue, studies on MT output quality in foreign language education remain scarce. Moreover, as MT is advancing every year, updated studies are imperative. Therefore, the present study investigated the quality of MT outputs (Google Translate) from Korean to English by comparing it with the English-translated texts of intermediate English as a foreign language students. The study also examined the factors within the source texts that affect the quality of MT outputs. Five trained evaluators examined multiple aspects of MT output samples (N = 104) and students' English texts (N = 104), including mechanics, vocabulary, grammar, and context. The results showed that both texts were equally comprehensible, but MT outperformed the students in most aspects under investigation. The study further found that only two factors in the source texts -- punctuation and sentence complexity -- influenced MT output quality, whereas lexical and grammatical accuracy, lexical diversity, and contextual understanding did not affect it. Based on the results, the study presents classroom implications for using MT for educational purposes.