Developing oral presentation skills requires both practice and expert feedback. Several systems have been developed during the last 20 years to provide ample practice opportunities and automated feedback for novice presenters. However, a comprehensive literature review discovered that none of those systems have been adequately evaluated in real learning settings. This work is the first randomised controlled evaluation of the impact that one of these systems has in developing oral presentation skills during a real semester-long learning activity with 180 students. The main findings are that (1) the development of different dimensions of the oral presentations are not affected equally by the automated feedback and (2) there is a small but statistically significant effect of the use of the tool when a subsequent presentation is evaluated by a human expert.