In this article we report on middle school students' understanding of concepts related to the second step of the statistical problem-solving process, study design: create a plan and collect data. Students participated in an open-ended statistics lesson involving nonexperimental study design. For this research we analyzed the students' discourse and categorized their statements using Groth's (2003) "Levels of Thinking Associated With Designing a Nonexperimental Study." Similar to studies by other researchers, we found middle school students: intuitively used a naïve conception of stratified random sampling as an appropriate method for collecting data; and, recognized some forms of bias. Based on these findings we offer recommendations for how to modify and implement statistics lessons so that students develop an understanding of study design, create a plan and collect data. Finally, we put forward suggestions for future research.