Background: Self-assessment serves to improve learning through timely feedback on one's solution and iterative refinement as a way to improve one's competence. However, the complexity of the self-assessment process is widely recognized, as well as that students can benefit from it only if their assessment is accurate enough. Objectives: In order to gain more insight into the self-assessment process we analysed students' behaviour, accuracy, and question-related characteristics that influence the capability of self-assessment in two studies. Methods: The initial study examined 131 undergraduate students using voluntary self-assessment questions in an online course in a B.Sc. Computer Science program while a year later a replication study with the same research settings was applied to a different cohort of 264 undergraduate students with minor modifications to the question design, in the light of the original findings. Results and Conclusions: Results from both studies show that similar patterns could be observed for usage and of accuracy and score distribution for almost all questions. Item difficulty and comprehensiveness of the sample solution were identified as features of self-assessment questions affecting student's self-assessment capability. The replication study showed that task design can be modified to affect students' accuracy. Recommendations to make self-assessment tasks effective and efficient for learning are provided.