Background: In recent years, there has been increased research interest in the functioning of working memory in people with intellectual disabilities. Although studies have repeatedly found these individuals to have weak working memory skills, few investigations have distinguished between different degrees of intellectual disability. This study aims to help close this research gap and, in so doing, to examine whether the deficits observed reflect a developmental lag or a qualitative deviation from normal development. Method: In a 5-group design, the working memory performance of a group of 15-year-olds with mild intellectual disability (IQ 50-69) was compared with that of two groups of children (aged 10 and 15 years) with borderline intellectual disability (IQ 70-84) and with that of two groups of children with average intellectual abilities (IQ 90-115) matched for mental and chronological age (aged 7 and 15 years). All children were administered a comprehensive battery of tests assessing the central executive, the visual-spatial sketchpad, and the phonological loop. Results: The results showed deficits in all three components of working memory, and revealed that these deficits increased with the degree of intellectual disability. The findings indicate that, relative to their mental age peers, children with learning difficulties show structural abnormalities in the phonological store of the phonological loop, but developmental lags in the other two subsystems. Conclusions: Similar patterns of results emerged for both subgroups of children with intellectual disability, indicating that problems with phonological information processing seem to be one of the causes of cognitive impairment in individuals with intellectual disability.