Technological tools that provide automated feedback on classroom teaching afford a unique opportunity for educators to engage in self-reflection and work towards improvement goals, in particular to ensure that their instructional environment is equitable and productive for students. More information is needed about how teachers experience automated professional learning tools, including what they perceive as relevant and impactful for their everyday teaching. This mixed-methods study explored the perceptions and engagement of 21 math teachers who used an AI-based tool that generates information about their discourse practices from classroom recordings. Findings indicate that teachers perceived the tool to have a high utility value, especially those who elected to use it over two school years. These teachers increased their use of talk moves over time, suggesting that they were making intentional changes due to their review and uptake of the personalized feedback. These results from this study speak to promising directions for developing AI-based professional learning tools that can support teacher learning and instructional improvement, particularly tools with robust perceived utility.