In this article, we explored the potential of the Student Mental Model Analyzer for Research and Teaching (SMART) technology that was designed to help students write and revise high-quality summaries of complex texts. Grounded in the design research, the aim of this pilot study was twofold: (a) to generate insights to further improve the SMART technology as a product and (b) to provide design knowledge that could guide the development and implementation of a formative assessment and feedback technology for reading comprehension. To that end, we conducted a formative evaluation of a beta version of the SMART technology in the context of a graduate-level asynchronous online course in which 21 students were enrolled. To investigate the potential of SMART and user experiences, we analyzed mixed data from the perspectives of the Technology Acceptance Model (i.e., ease of use and usefulness) and the quality aspects of learning technology (i.e., content, interface, and support). Results indicate that students improved their reading comprehension through SMART feedback and valued their use of SMART. The findings of the study also suggest design improvements for SMART. Furthermore, this research highlights the design approaches to formative assessment and feedback tools, focusing on improving users' technology acceptance.