Childhood adversity can lead to cognitive deficits or enhancements, depending on many factors. Though progress has been made, two challenges prevent us from integrating and better understanding these patterns. First, studies commonly use and interpret raw performance differences, such as response times, which conflate different stages of cognitive processing. Second, most studies either isolate or aggregate abilities, obscuring the degree to which individual differences reflect task-general (shared) or task-specific (unique) processes. We addressed these challenges using Drift Diffusion Modeling (DDM) and structural equation modeling (SEM). Leveraging a large, representative sample of 9--10 year-olds from the Adolescent Brain Cognitive Development (ABCD) study, we examined how two forms of adversity--material deprivation and household threat--were associated with performance on tasks measuring processing speed, inhibition, attention shifting, and mental rotation. Using DDM, we decomposed performance on each task into three distinct stages of processing: speed of information uptake, response caution, and stimulus encoding/response execution. Using SEM, we isolated task-general and task-specific variances in each processing stage and estimated their associations with the two forms of adversity. Youth with more exposure to household threat (but not material deprivation) showed slower task-general processing speed, but showed intact task-specific abilities. In addition, youth with more exposure to household threat tended to respond more cautiously in general. These findings suggest that traditional assessments might overestimate the extent to which childhood adversity reduces specific abilities. By combining DDM and SEM approaches, we can develop a more nuanced understanding of how adversity affects different aspects of youth's cognitive performance.