Researchers design diagnostic assessments to measure students' knowledge structures and processing skills to provide information about their cognitive attribute. The purpose of this study is to determine the instrument's validity and score reliability, as well as to investigate the use of classical test theory to identify item characteristics. The data used in the form of responses to elementary school mathematics subject. There are 166 students from 5 public elementary schools who participate in the study. The data analysis technique used is the analysis of item characteristics based on classical test theory using the R software package. The results showed that the developed mathematical ability diagnostic instrument had high content validity based on the Aiken formula and valid construct validity based on the CFA approach. According to the Spearman-Brown formulation, the correlation coefficient is about 0.889, indicating high internal consistency reliability. In the index of difficulty level, overall, it is categorized as moderate items. The discriminatory index shows that there are two items, namely items 9 and 17, with low discriminating power, so the two items are not used. Of the 60 total distractors, 5 (8.3%) did not function well because less than 5% of the participants chose them. In contrast, as many as 55 distractors (91.7%) have functioned well.