In the Next Generation Science Standards (NGSS), problem-solving skills are part of science and engineering practices for K-12 students in the United States. Evaluating these skills for the youngest learners is difficult due to the lack of established measures. This paper reports on our process of developing an observation instrument to measure preschool children's learning and their application of problem-solving skills, namely, the steps of the engineering design process (EDP). The instrument, Engineering Preschool Children Observation Tool (EPCOT), was intended to evaluate the frequencies of problem-solving behaviors and use of EDP-related vocabulary by observing preschoolers engaged with the Seeds of STEM eight-unit curriculum in the classroom. In this paper, we describe the development process and revision of EPCOT, its current constructs, and present descriptive findings from using the tool in a pilot study with sixteen classrooms: eight intervention classrooms who received the entire curriculum, and eight comparison classrooms who received only the eighth unit of the curriculum (to enable comparison). We found that, out of 34 possible behaviors across the problem-solving process, children in all classrooms engaged in 31 unique problem-solving behaviors, suggesting that preschool children are indeed capable of meaningfully engaging in solving problems. We also observed a trend that children who were exposed to more of the curriculum (the intervention group) produced more novel vocabulary words than those in the comparison group, who tended to repeat vocabulary words. Since EPCOT was developed in alignment with state and national standards, we believe it has the potential to be used with other early childhood engineering/problem-solving curricula.