Purpose: Some degree programs in colleges and universities utilize entrance exams to ensure that students pursuing a given degree have mastered foundational concepts needed for that program. However, often these exams become a barrier to student success. The purpose of this study is to discuss the impact of policies governing an undergraduate Computer Science (CS) entry/qualifying exam at a large public university in the United States on overall student success in the program. This case study focuses on whether reforming program policies impacts students' time-to-degree, graduation and mastery in CS core skills. Design/methodology/approach: This case study describes how the CS student success was improved by updating program policies based on institutional data and the input of course instructors. The policy changes include introducing a maximum limit to attempt the exam, changing the exam requirements as well as the structure of the exam itself. Findings: The pass rates of students taking this qualifying exam were significantly and consistently low over the years. Students who were potentially a better fit for programs other than CS delayed their start in those other programs by taking and failing to pass the CS qualifying exam multiple times. As a result of implementing new CS program policies, many more students attempted the exam at an appropriate time. Also, they graduated on time in the years 2016-2018. Originality/value: This paper presents a case study in which the graduation rates and time to degree of a CS program with a qualifying exam were improved after a policy intervention. The results demonstrate that simple policy changes can improve student success. Findings from this study may guide other programs with similar characteristics to improve their student success.