Background: High rates of conduct difficulties have been reported among children with borderline intellectual disabilities or intellectual disabilities. Little is known about predictors of the persistence of conduct difficulties in the pre-adolescent period in this high-risk group. Methods: Secondary analysis of data from the first three waves of the Longitudinal Study of Australian Children. Results: The prevalence of conduct difficulties at age 4/5 years among children with cognitive delay was independently associated with: living in an income poor household; living in a more deprived neighbourhood; having co-morbid emotional difficulties; having poorer pro-social behaviours; poorer maternal health; greater exposure to angry/harsh parenting; and greater exposure to more inconsistent parenting. The persistence of conduct difficulties at ages 6/7 years and 8/9 years was independently associated with: male child gender; living in material hardship; child expressive communication difficulties; living in a single parent headed household; and greater exposure to angry/harsh and inconsistent parenting. When compared to their more intellectually able peers, children with borderline or intellectual disability were significantly more likely to exhibit persistent conduct difficulties, but only when exposed to multiple environmental risks. These results were consistent across different thresholds for measuring conduct difficulties and different thresholds for measuring cognitive delay. Conclusions: The persistence and high levels of conduct difficulties among this high risk group appears to be associated with a combination of increased risk of exposure to environmental adversity and decreased resilience when so exposed.