Data comparison problems are used in teaching and science education research that focuses on students' ability to compare datasets and their conceptual understanding of measurement uncertainties. However, the evaluation of students' decisions in these problems can pose a problem: e.g., students making a correct decision for the wrong reasons. Three previous studies, that share the same context and data comparison problem but where participants had increasing conceptual knowledge of measurement uncertainties, are revisited. The comparison shows a troublesome result: increasing conceptual knowledge does not lead to better decision making in the data comparison problem. In this research, we have looked into this apparent discrepancy by comparing and reanalyzing the data from these three studies. We have analyzed students' justifications by coding them based on the compared quantity and the deciding criterion, giving a highly detailed insight into what they do when comparing the datasets. The results show clear differences in the quality of the justifications across the studies and by combining the results with the decisions, we could successfully identify four cases of correct and incorrect decisions for right or wrong reasons. This analysis showed a high prevalence of correct decisions for wrong reasons in two of the studies, resolving the discrepancy in the initial comparison of these studies. The implication of our analysis is that simply asking students to make a decision in data comparison problems is not a suitable probe to gauge their ability to compare datasets or their conceptual understanding of measurement uncertainties and a probe like this should always be complemented by an analysis of the justification.