Scientific reasoning ability (SRA) is widely recognized as an essential goal for science education. There is much discussion on the design and development of assessment frameworks as viable tools to foster SRA. However, established assessments mostly focus on the level of students reasoning attainment. Student ability to use evidence to support reasoning is not adequately addressed and evaluated. In this study, the 6-level SRA assessment framework was conceptualized and validated iteratively via synthesizing literature and a Delphi study. Guided by the framework, an SRA assessment tool adopting and adapting PISA test items and self-created items was developed and administered to 593 secondary students (including 318 8th Graders and 275 9th Graders) in mainland China. Pearson correlation analysis of SRA assessment score and their scores in scientific reasoning provided criterion-related validation for the former (Pearson correlation = 0.527). Rasch analysis conducted further confirmed the validity and reliability of the SRA test and the assessment framework. Combing quantitative and qualitative methods, the study provides a valid and reliable analytical framework of SRA. It can inform the design of SRA assessments in various science education contexts for diversified audiences.