Background: Academic dishonesty (AD) and trustworthy assessment (TA) are fundamental issues in the context of an online assessment. However, little systematic work currently exists on how researchers have explored AD and TA issues in online assessment practice. Objectives: Hence, this research aimed at investigating the latest findings regarding AD forms, factors affecting AD and TA, and solutions to reduce AD and increase TA to maintain the quality of online assessment. Methods: We reviewed 52 articles in Scopus and Web of Science databases from January 2017 to April 2021 using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses model as a guideline to perform a systematic literature review that included three stages, namely planning, conducting, and reporting. Results and conclusions: Our review found that there were different forms of AD among students in online learning namely plagiarism, cheating, collusion, and using jockeys. Individual factors such as being lazy to learn, lack of ability, and poor awareness as well as situational factors including the influence of friends, the pressure of the courses, and ease of access to information were strongly associated with AD. A technology-based approach such as using plagiarism-checking software, multi-artificial intelligence (AI) in a learning management system, computer adaptive tests, and online proctoring as well as pedagogical-based approaches, such as implementing a research ethics course programme, and a re-design assessment form such as oral-based and dynamic assessment to reduce cheating behaviour and also sociocultural and sociotechnical adjustment related to the online assessment are reported to reduce AD and increase TA. Implications: Educators should adjust the design of online learning and assessment methods as soon as possible. The identified gaps point towards unexplored study on AI, machine learning, learning analytics tools, and related issues of AD and TA in K12 education could motivated future work in the field.