Online metacognitive skills are the real-time awareness of cognition, which can effectively promote science learning and improve performance in solving scientific problems. Therefore, it is important to enhance and diagnose students' online metacognitive skills in science education. This study aimed to evaluate ninth-grade students' online metacognitive skills while processing chemistry problems. To achieve this goal, this study constructed a framework for guiding the development of an instrument comprising 12 two-tier items. A total of 258 ninth graders took part in the field testing in Jiangsu, China. A partial credit Rasch model analysis was employed to inform instrument development and evaluation. The results revealed that this instrument was valid and reliable for assessing students' online metacognitive skills. Nearly 60% of the ninth-grade students in this sample were able to monitor their own thought processes or evaluate their own cognitive performance in processing chemistry problems. About one-third of the students could regulate their thought processes. However, less than 4% of the students could make attributions about their cognitive performance.