Medical schools all over the world select applicants using non-cognitive and cognitive criteria. The predictive value of these different types of selection criteria has however never been investigated within the same curriculum while using a control group. We therefore set up a study that enabled us to compare the academic performance of three different admission groups, all composed of school-leaver entry students, and all enrolled in the same Bachelor curriculum: students selected on non-cognitive criteria, students selected on cognitive criteria and students admitted by lottery. First-year GPA and number of course credits (ECTS) at 52 weeks after enrollment of non-cognitive selected students (N = 102), cognitive selected students (N = 92) and lottery-admitted students (N = 356) were analyzed. In addition, chances of dropping out, probability of passing the third-year OSCE, and completing the Bachelor program in 3 years were compared. Although there were no significant differences between the admission groups in first-year GPA, cognitive selected students had obtained significantly more ECTS at 52 weeks and dropped out less often than lottery-admitted students. Probabilities of passing the OSCE and completing the bachelor program in 3 years did not significantly differ between the groups. These findings indicate that the use of only non-cognitive selection criteria is not sufficient to select the best academically performing students, most probably because a minimal cognitive basis is needed to succeed in medical school.