We present lessons learned from an ongoing attempt to conceptualize, develop, and refine a way for teachers to gather formative assessment evidence about classroom argumentation as it happens. The system--named DiALoG (Diagnosing Argumentation Levels of Groups)--includes a digital scoring tool that allows teachers to assess oral classroom argumentation across two primary dimensions: one to capture the Intrapersonal, discipline-specific features of scientific arguments, and another to capture the Interpersonal, group regulatory features of argumentation as a dynamic social act. Coupled with the digital assessment are responsive mini-lessons (RMLs), which provide follow-up curriculum for teachers to respond to different levels of classroom argumentation proficiency for each item assessed. We use classroom observations, interviews, and surveys from piloting science teachers in two different states to iteratively refine this multifaceted formative assessment system of oral classroom argumentation. Lessons learned include the realization by pilot teachers that using the DiALoG system fine-tunes their "professional vision" to notice student practices they had not previously considered, and the accompanying RMLs help fill gaps in their pedagogical content knowledge and repertoire. Furthermore, while the DiALoG system is intended to be a formative assessment, we learned that the mere presence of numerical scores can queue teacher schema for summative assessment. This prompted us to do away with numbers entirely in the latest version of our digital scoring tool. Such lessons learned from teacher experiences with the development of a novel formative assessment system like DiALoG can be instructive to the development of science educational technology more broadly.