Purpose: This prospective longitudinal study aimed to explore (a) the development of social-pragmatic understanding of children with bilateral hearing aids (BiHAs), bilateral cochlear implants (BiCIs), and typical hearing (TH) between the ages of 4 and 6 years and (b) group differences between children with BiHAs, BiCIs, and TH. Method: The Pragma test was used for a comprehensive assessment of social-pragmatic understanding of a total of 86 children: 19 children with BiHAs, 22 children with BiCIs, and 45 children with TH. The Pragma test requires answering socially and contextually demanding questions and explaining the right answers. The explanation tasks are targeted at studying the participant's own awareness of the inferencing process. The children in this study were assessed yearly at the ages of 4, 5, and 6 years. Results: The participants with BiHAs, BiCIs, and TH showed significant development in their social-pragmatic understanding between the ages of 4 and 6 years, but most children with hearing loss (HL) still did not meet age expectations at the age of 6 years. Children with BiHAs and BiCIs both showed large-scale inferential difficulties, including utilizing theory of mind, utilizing verbal and visual information, and understanding conversational norms and emotions in context. Conclusions: Children with BiHAs and BiCIs are at risk of delays in social-pragmatic understanding despite early detection of HL, early amplification, and cochlear implantation. Therefore, the social-pragmatic abilities of children with HL should be assessed regularly, and the children with HL should have early access to social-pragmatic interventions where utilizing contextual information is practiced comprehensively.