Increasing student engagement through Electronic Response Systems (clickers) has been widely researched. Its success largely depends on the quality of multiple-choice questions used by instructors. This paper describes a pilot project that focused on the implementation of online collaborative multiple-choice question repository, PeerWise, in a Physics Methods course for secondary teachers at a large Canadian Teacher Education program. We investigated different facets of PeerWise implementation, teacher-candidates' engagement through the system, and how PeerWise can support teacher-candidates' collaboration on designing high quality conceptual multiple-choice questions. This study sheds light on the results of this implementation and suggests directions for future research.