With recent improvements in machine translation (MT) accuracy, MT has gained unprecedented popularity in second language (L2) learning. Despite the significant number of studies on MT use, the effects of using MT on students' retention of learning or secondary school students' use of MT in L2 writing has rarely been researched. The current study investigates the effectiveness of using machine translation on Korean middle school students' L2 writing over an extended period of time. This study evaluated the complexity, accuracy, lexical diversity, and fluency of four versions of the students' writing (pretest, MT-assisted version, posttest, and 2-week or 4-week retention tests) and measured errors in punctuation, spelling, vocabulary, and grammar in each version. The study employed a quantitative research method including descriptive statistics, repeated measures ANOVA, and paired t tests. The results showed that fluency, accuracy, and complexity significantly increased in the MT-assisted version in every aspect of writing, but decreased in the subsequent versions without MT (post- and retention tests). Decreases occurred more frequently with grammatical items than with lexical items. Despite the decreases, all of the items measured in the study scored higher in the retention tests than in the pretest, which indicates that the use of MT had a positive effect on L2 writing.