To meet accreditation expectations, teacher preparation programs must demonstrate their candidates are evaluated using summative assessment tools that yield sound, reliable, and valid data. These tools are primarily used by the clinical experience team -- university supervisors and mentor teachers. Institutional beliefs regarding best practices and external stakeholder demands often influence the design, execution, and evaluation of these tools, and the scholarly literature is robust with studies of the design and execution of the tools. This study extends prior research to investigate, in a novel way, evaluation of the tool and discrepancies among raters in the data yielded by the instrument. How important is interrater agreement on summative assessment tools in the student teaching experience? Should we simply dismiss these differences, forge ahead to minimize disagreement, or seek to better understand how and why limited agreement might occur? In this paper, we explore methods to evaluate interrater agreement between mentors and university supervisors and the ways in which these data can inform purposeful continuous programmatic improvement. Furthermore, we contend that "less is more," as our data findings reveal similar outcomes with both simple and more complex statistical methods of examining agreement.