Autism spectrum disorder (ASD) and attention deficit hyperactivity disorder (ADHD) are highly heritable neurodevelopmental disorders that frequently co-occur. Both rare and common genetic variants are important for ASD and ADHD risk but their combined contribution to clinical heterogeneity is unclear. In a sample of 39 ASD and/or ADHD families we estimated the overall variance explained by known rare copy number variants (CNVs) and polygenic risk score (PRS) from common variants to be 10% in comorbid ASD/ADHD, 4% in ASD and 2% in ADHD. We show that burden of large, rare CNVs and PRS is significantly higher in adult ASD and/or ADHD patients with sustained need for specialist care compared to their unaffected relatives, while affected relatives fall in-between the two.