No records of systematic reviews focused on deep learning in open learning have been found, although there has been some focus on other areas of machine learning. Through a systematic review, this study aimed to determine the trends, applied computational techniques, and areas of educational use of deep learning in open learning. The PRISMA protocol was used, and the Web of Science Core Collection (2019-2023) was searched. VOSviewer was used for networking and clustering, and in-depth analysis was employed to answer the research questions. Among the main results, it is worth noting that the scientific literature has focused on the following areas: (a) predicting student dropout, (b) automatic grading of short answers, and (c) recommending MOOC courses. It was concluded that pedagogical challenges have included the effective personalization of content for different learning styles and the need to address possible inherent biases in the datasets (e.g., socio-demographics, traces, competencies, learning objectives) used for training. Regarding deep learning, we observed an increase in the use of pre-trained models, the development of more efficient architectures, and the growing use of interpretability techniques. Technological challenges related to the use of large datasets, intensive computation, interpretability, knowledge transfer, ethics and bias, security, and cost of implementation were also evident.