A little less than half of the students of higher Ed institutions in the US graduate in four years, and only around 60% finish in six years. Retention rates are also less than ideal. Colleges have been experimenting with a variety of programs and policies to address this issue, especially less selective institutions whose rates are significantly lower. In this paper, we evaluate a student success and retention program called Early Alert that was implemented at a public state university in the US with a medium-to-large student body. Our dataset contains several years' worth of information on students' socio-demographic characteristics, class standing and average grades (GPAs), as well as their midterm and final grades in undergraduate courses. We employ several causal inference techniques developed for observational studies and elicit negative average treatment effects on the treated (ATT). Since it is conceivable that unobserved confounders are the real drivers of our empirical results, not the treatment, we carry out two different types of sensitivity analyses. Together with our treatment effect estimations, they lead us to the main conclusion that Early Alert does not improve student performance, at least not in the short run (as measured by course performance), and likely has a negligible impact.