In educational reform initiatives worldwide and in Singapore, educators are increasingly encouraged to take ownership of their work through involvement in practitioner inquiry and evaluation efforts to transform teaching and learning. Educators' epistemological beliefs -- their view of knowledge and knowing -- play a critical role as they learn to become critical, reflective actors in the knowledge construction endeavour. This mixed-methods study investigates whether and how educators' epistemological beliefs influence their understanding of programme evaluation as viewed through participants' journey of learning to design a real-world evaluation study in the context of Masters-level coursework. Findings suggest that participants' epistemological scale scores are associated with their learning outcomes as manifested in the quality of their evaluation plan. Specifically, participants with higher or lower scores show important differences in their view of evaluation and what a good evaluation requires, their relationship with knowledge about programme evaluation, the nature of challenges encountered in designing their evaluation plans, and how they navigate their challenges. Findings have implications for purposes and approaches of teacher professional learning and offer insights into the re-design of learning opportunities to transform educators' epistemological beliefs in general and in relation to discipline-specific learning.