This article uses Rasch analysis to explore the efficacy of a questionnaire designed to assist university teaching staff in identifying those Level 4 students most in need of mathematics support. The students were all taking a mathematics module as part of their first year Computing curriculum, and the questionnaire explores the students' previous mathematical experience and asks them to evaluate their basic skills using Likert scale responses. Analysis of the 124 student responses identifies those questionnaire items that do not fit the Rasch model well and indicates some areas where further improvements to the questionnaire can be made.