This study aims to evaluate students' ability to process the context information embedded in chemistry problems. To achieve this goal, a diagnostic measurement instrument was developed, comprising 28 short-answer items embedded in seven context-based chemistry tasks. Four hundred and ninety-three ninth-graders took part in the testing in Jiangsu, China. The partial credit Rasch model was applied to establish evidence of validity and reliability of the measurement instrument. Results showed that this instrument could produce reliable and valid measures of students' context-based chemistry problem-solving skills. Nearly half of the ninth-grade students had a high level of context extracting skills and a moderate level of context integrating skills. However, most ninth-graders only had a low level of context reasoning or argumentation skills. In addition, most students had unbalanced skill levels, and their skill levels decreased or equaled as the complexity of skill components increased. This study provides insights into the extent of students' skills of dealing with context information in chemistry problems by developing a reliable and valid measurement instrument. The findings of this study might inform chemistry teachers to improve their teaching practices, call attention to test developers in taking much care of designing the context, and enable further studies to relate context-based problem-solving skills to chemistry concept learning and application.