This essay presents the reflections of four Education Doctorate (EdD) students on the pilot testing strategies used during an online research methods course. Rigorous questionnaire and interview development skills are challenging to acquire. Pilot testing is an under-researched stage of instrument design, yet it is crucial to ensure validity and reliability, reduce bias, and psychologically prepare researchers for data collection. A structured, multi-step pilot testing process led to the collective development of stronger scholar-practitioner identities, the use of innovative synchronous/asynchronous methods during COVID-19 and increased academic rigor. These reflections demonstrate how several types of pilot testing can support the development of rigorous data collection instruments and prepare post-graduate students for the psychological and technical challenges they may encounter in future research.