Early impaired response to social partners' distress may negatively impact subsequent social development. Identifying factors contributing to successful responding may inform assessment and intervention. This study explores how: (1) social impairment, and (2) partner familiarity relate to response to partners' distress. Infants with and without older siblings with ASD were assessed at 12 (n = 29) and 15 (n = 35) months for social impairment markers, and responses to mother and experimenter each feigning distress. Infants with more social impairment showed less attention and affect at 15, but not 12 months. Infants attended more to the unfamiliar person, but exhibited greater affect toward the familiar person at 12 months. Results revealed social impairment and familiarity were separately related to infant response to partners' distress.