We provide evidence of validity for a newly developed diagnostic competence model of operation sense, by both (a) describing the theoretically substantiated development of the competence model in close association with its use within a large-scale formative assessment and (b) providing empirical evidence for the theoretically described cognitive levels of competences. The competence model describes students' operation sense on four distinct levels. On each level, the model elaborates on the characteristics of tasks that students on this level are able to answer correctly. Moreover, the model explains this by referring to two kinds of cognitive processes that are supposed to be necessary to respond to these kinds of tasks successfully. In a validation study, about 85% of the variance in the item difficulties was explained by the four, a priori allocated, levels of operation sense. We discuss the relevance of the validation of the diagnostic competence model for the provision of criterion-referenced feedback in a large-scale formative assessment, including suggestions for teachers' subsequent support activities, and the contributions of the model to the state of research about operation sense.