The aim of this study is to understand how students' mathematical thinking is activated and nurtured in solving a modeling problem, where the problem situation involves the design of a system. From a STEM integrated perspective, 9th grade students worked on a modeling task aiming to create an identification system based on hand biometrics. The theoretical framework proposes a conceptualization of the interplay between the mathematical modeling process, from a cognitive perspective, and the engineering design process. Central ideas refer to the cyclical nature of both processes and to the sub-processes involved in them. The empirical data were collected in two design-based research cycles with different 9th grade classes. The data from the groups' audio and video recording and the students' productions were analyzed under a directed qualitative content analysis informed by theory. The results showed a global pattern in the students' thinking in solving a design system problem. The overlapping and interplay between the mathematical modeling and the design process was a prominent characteristic of students' thinking. The modeling cycle was mirrored by a design cycle, with both running in parallel. System thinking pushed and drove students' mathematical thinking, from the system requirements to the prototype validation.