This study focused on the development and psychometric evaluation of the 16-item Critical Musicality Scale (CMS). The participants in the study comprised (N = 118) secondary students (mostly 13 to 14 years old) from Singapore. Four factors of critical musicality were hypothesised and empirically tested. The full sample was used to conduct an exploratory factor analysis which affirmed the hypothesised factor structure of CMS. Close to 85% of the full sample was randomly selected (n = 93) and associated data set was used to conduct confirmatory factor analysis. The four-factor structure of the items making up CMS was found to have a better fit to the data compared to the one-factor and three-factor models. The final four-factor model was found to be invariant across gender and was able to explain about 51% to 74% of the variance in the data. The reliability of each factor of critical musicality was found to be satisfactory. As additional evidence for the construct validity of the scale, it was found that all of the dimensions of critical musicality had significant positive correlations with the students' behavioural and emotional engagement in music learning activities.