High dropout rate exists universally in massive open online courses (MOOCs) due to the separation of teachers and learners in space and time. Dropout prediction using the machine learning method is an extremely important prerequisite to identify potential at-risk learners to improve learning. It has attracted much attention and there have emerged a few reviews. However, current reviews of MOOC dropout prediction exist some common limitations. Firstly, different definitions of course dropout are not summarized. Secondly, there lacks an overall framework of MOOC dropout prediction. Thirdly, some key challenges are not fully explored. Thus, unlike past reviews, this systematic review concludes with three categories of definitions of course dropout. Then it proposes an overall framework including factors affecting dropout, general feature extraction methods, various machine learning methods and evaluation methods. Finally, the key challenges of interpretability, imbalanced data and semantic learning trajectory modeling are proposed. This study aims to enable researchers to capture a whole picture of dropout prediction from the perspective of machine learning.