Much of the focus on learning technologies has been on structuring innovative learning experiences and on managing distance and hybrid learning environments. This article focuses on the use of technology as an important formative assessment and feedback tool. The rationale for this focus is based on prior research findings that suggest that timely and informative feedback promotes learning. The general purpose of this article is to promote a focus on formative assessment, especially with challenging problems so as to help develop critical thinking skills. This is not a meta-analysis although we encourage such studies so as to emphasis the role that formative assessment plays in supporting learning. Much of the prior research on formative assessment has not involved advanced digital technologies and very little of that research has focused on complex and challenging problem-solving tasks. We review prior work on the use of problem conceptualizations elicited during problem solving activities and on stealth assessments of learner choices and decisions in online activities. We present a conceptual framework based on prior research and theory for conducting formative assessments in real-time with regard to complex problem-solving tasks. We then present an elaboration of how formative assessment can be used to support learning a common intellectual skill involving a discrimination task and to develop an appropriate cognitive strategy for that kind of problem. We conclude with recommendations for further research on the use of technology in support of formative assessment.