There is a growing interest in using comparative judgement as a peer assessment tool involving students choosing the 'better' of two pieces of work. However, it has not been shown whether peer assessment through comparative judgement is effective for students' learning. To gain a better understanding of the comparative judgement process, we investigated how students form pairwise judgements in a controlled setting. We analysed undergraduate students' think-aloud comments while they compared other students' mathematics solutions. The criteria students used to inform their pairwise choices included, how easy a solution was to follow, whether it was accurate, which method was used, and how it was presented. These choice criteria were found to be hierarchical in nature where students typically drew on a primary set of criteria to help form a judgement. When this failed to differentiate between two solutions, students continued down their list of criteria until a judgement was formed.