One goal of higher education is the development of students' evaluative judgement, that is, the capacity to make judgements of the quality of work done by oneself and others. Supporting students to critique the work of peers provides opportunities for evaluative judgement, which we explored using an online platform, PeerWise, in a second year genetics course. PeerWise enables students to write, comment on, answer and rate multiple choice questions. Our study aimed to investigate students' understanding of question quality by identifying the criteria they used to assess questions and how these were applied to make judgements. Students' comments on questions, in which they provided an opinion with a reason, were analysed. This analysis was done in two stages; an initial qualitative analysis of comments from one cohort from which the criteria emerged, and validation and quantification of criteria usage in an independent cohort. We identified five criteria that were used to assess questions: knowledge, clarity and presentation, complexity, engagement and explanation. We then analysed how the criteria were used to show that students demonstrated evaluative judgement, not only by generating reasonable criteria, but also by making complex judgements about question quality invoking multiple criteria and constructive suggestions for improvement.