Background: Learners' writing skills are critical to their academic and professional development. Previous studies have shown that learners' self-assessment during writing is essential for assessing their writing products and monitoring their writing processes. However, conducting practical self-assessments of writing remains challenging for learners without help, such as formative feedback. Objectives: To facilitate learners' self-assessment in writing, we developed a writing analytics toolkit and used data visualisation and cutting-edge machine learning technology that provides real-time and formative feedback to learners. Methods: To investigate whether our newly-developed tool affects the accuracy and process of learners' self-assessment, we conducted a lab study. We assigned 59 learners to complete writing (2 h) and revising (1 h) tasks. During the revision stage, we randomly assigned the learners to two groups: one group used the writing analytics toolkit while the second group was not granted access to the toolkit. Learners' self-assessment accuracy and process of self-assessment were compared between the two groups. Results: In our study, we found the toolkit helped learners in the experimental group improve the self-assessment accuracy of their writing products compared to the learners in the control group. In addition, we also found that the affordances of the toolkit affected the learners' self-assessment process, and poor design affordances may have prevented the learners from reflecting by themselves. Conclusions: Together, our empirical study shed light on the design of future writing analytics tools which aim at improving learners' self-assessment during formative writing processes.