The aim of the research is to investigate the academic achievement of ChatGPT, an artificial intelligence based chatbot, in a national mathematics exam. For this purpose, 3.5 and 4 versions of ChatGPT were asked mathematics questions in a national exam. The method of the research is a case study. In the research, 3.5 and 4 versions of ChatGPT were used as data collection tools. The answers given by both versions of ChatGPT were analyzed separately by three researchers. As a result of the analysis of the data, it was found that ChatGPT-4 was more successful in the exam compared to ChatGPT-3,5 version, was better at understanding the questions asked, understood the instructions better and included more details in the question solution, and at the same time, both versions made common and different mistakes. According to the findings of the study, it was concluded that ChatGPT sometimes worked very well, sometimes only worked well and sometimes failed. In the light of the findings of the study, it can be suggested to use ChatGPT versions in mathematics education to obtain basic information and to get supervised help.