High-stakes testing (HST) is critical in the educational system of East-Asian countries like China, and the ongoing standards-based reform in many Western countries like the USA. HST has been criticized for corrupting an educational system through means like disengaging students from authentic learning. It seems that students' affective learning would be promoted if HST were abandoned. In this study, we used the mixed-method approach to investigate this hypothetical question by comparing the physics identity of high-school students in an HST-oriented educational context and college students who have transferred from an HST-oriented context to an almost HST-free one in China. We administered a physics identity survey with 108 students from two physics classes in a key high school and 233 undergraduate physics majors in a top-tier university who were separated by Gaokao as the most life-shaping test in China. We also interviewed sample participants about their physics learning. The data show the freshmen in college had a significantly higher average physics identity than the high-school students did. In the undergraduate physics program, the average physics identity had a deteriorating pattern from the freshmen, sophomores, to juniors. From high school to college, the students' intention of pursuing physics as a major or career was similarly low. Differently, the students' perception of lecturing changed, so did their self-discipline. Together, the findings suggested that simply discarding HST would not promote students' affective learning. An alternative was needed to replace the extrinsic pressure from HST, which could be either students' intrinsic engine of learning or another rewarding or coercive system.