Due to the learning paradox, students cannot have real difficulty in understanding a mathematical concept that they have not yet understood. There is a gap between real difficulties, directly experienced by students, and illusionary ones, only observed by researchers. This paper aims to offer a critical reflection on our understanding of the term "difficulty" in mathematics education research. We start this paper by arguing that a constructivist perspective, which has often been adopted in researches on mathematical task design, can deal with difficulties in solving a mathematical problem, but it cannot theoretically deal with those in understanding a mathematical concept. Therefore, we need the alternative philosophy of Robert Brandom's inferentialism to capture students' real difficulties in conceptual learning. From an inferentialist perspective, we introduce the idea of illusionary and real difficulties. The former is defined as what students cannot do, but they are not conscious of what they should do, while the latter is defined as what students cannot do despite their consciousness of what they should do. Through an eighth grade classroom episode, we argue that it is important in mathematics education research to focus not only on illusionary difficulties but also on the transition from illusionary to real difficulties. Researchers are encouraged to design a learning environment in which students become conscious of what they cannot do and to observe their mathematics learning in such an environment.