Having students write reflections has been shown to help teachers improve their instruction and students improve their learning outcomes. With the aid of Natural Language Processing (NLP), real-time educational applications that can assess and provide feedback on reflection quality can be deployed. In this work, we first evaluate various NLP approaches for developing a reflection quality prediction model, aiming to find a configuration that balances model simplicity and generalizability across courses. Second, using the model that best balances runtime performance and predictive accuracy, we evaluate the impact of using this model to trigger real-time feedback regarding reflection quality in a mobile application currently being deployed in multiple courses across universities. Analysis of students' long-term (semester-level) and short-term (reflection writing level) changes in reflection quality across multiple classes demonstrate the utility of the deployed model in encouraging students to submit reflections with higher quality. [This paper was published in: "AIED 2022, LNCS 13355," edited by M. M. Rodrigo et al., Springer Nature Switzerland AG, 2022, pp. 519-525.]