How do we make judgements about the quality of online courses? Checklists and rubrics are commonplace in higher education for establishing and measuring design features of online courses. They are created and used by institutions, academics and educational designers to standardise measures for quality online course design. Despite an intensifying spotlight on quality learning and teaching in higher education, no large-scale review of course quality instruments has occurred. This scoping review aimed to ascertain the conceptions of quality being promoted by course quality evaluation instruments and the capability-building resources that underpin these instruments. Seventy-five instruments used to measure quality in online course design in higher education were identified via a systematic search. These instruments were charted and coded. This paper reports on findings that summarise the key attributes of the course quality evaluation instruments, conceptualises a shared definition of course quality and proposes specific core criteria for measuring course quality under the domains of learning design, assessment and evaluation, usability and accessibility, social interaction and technology. The findings indicate that course quality instruments and their support resources have capability-building potential; yet, mostly this potential is not fully realised. We recommend shifting the capability-building potential of these tools from checking compliance to enabling skills development.