Over half of the children in the U.S. experience adversity early in childhood. These experiences, along with conditions in their families and neighborhoods, have profound developmental effects. The bioecological model of development includes these proximal contexts in a theory of development that incorporates the threats and supports present in these spaces to describe child development. This study used structural equation modeling to build latent measures of childhood adversity, family conflict, and neighborhood quality and tested theoretically-implied pathways to determine the relationships among these measures and cognitive outcomes in children. This study of US children ages 5-17 (N = 2,907) employed a nationally representative sample from the Panel Study of Income Dynamics to create and test these measures. Results indicate that adversity, family conflict, and a lack of neighborhood quality negatively impact cognitive function, even when controls for socioeconomic status and race are introduced. Testing of models indicated that family conflict and neighborhood quality are mediated by adverse childhood experiences, and these contexts should not be related to cognitive outcomes without the inclusion of adversity measures. This study provides further insight into the relationships among these contexts and children's lives, and offers guidance for future research with these constructs.