Background: The introduction of the Australian Research Training Scheme has been a strong reason for assuring the quality of the research higher degree (RHD) experience; if students experience poor supervision, an unsupportive climate, and inadequate infrastructure, prior research suggests RHD students will be less likely to complete their degree, with negative consequences for the student, the university, and society at large. Aims: The present study examines the psychometric properties of a survey instrument, the Student Research Experience Questionnaire (SREQ), for measuring the RHD experience of currently enrolled students. The core scales of the SREQ focus on student experiences of Supervision; Infrastructure; Intellectual and Social Climate; and Generic Skills Development. Sample: Participants were 2,213 postgraduate research students of a large, research-intensive Australian university. Results: Preliminary factor analyses conducted at the student level supported the "a priori" four factors that the SREQ was designed to measure. However, multi-level analyses indicated that there was almost no differentiation between faculties or departments nested with faculties, suggesting that the SREQ responses are not appropriate for benchmarking faculties or departments. Conclusions: Consistent with earlier research based on comparisons across universities, the SREQ is shown to be almost completely unreliable in terms of benchmarking faculties or departments within a university.