Previous research documented positive associations between cognitively stimulating work and levels of cognitive performance, while longitudinal associations are less clear. We used 20-year longitudinal data from the Interdisciplinary Longitudinal Study of Adult Development and Aging (ILSE) to examine the role of perceived work environment (autonomy, innovation, social integration, and stress) and work activities derived from an occupational database (related to information, to people, and manual activities) for trajectories of cognitive abilities (processing speed, fluid and crystallized intelligence). We used data from 374 participants in the ILSE midlife cohort (born 1950-52) who were working at baseline and had valid observations on work characteristics and control variables including education, gender, region (former East vs. West Germany), and personal income (mean age at baseline = 44 years, SD = 1, 44% women). Cognitively stimulating perceived work environments (higher levels of autonomy and innovation), higher levels of work activities related to information and people, and lower levels of manual activity at baseline were related to higher initial levels of cognitive ability. Higher work stress was related to higher baseline fluid ability. These associations were largely not independent of control variables. Higher social integration at work was related to less steep increase in crystallized intelligence and higher work stress was related to less decline in processing speed. In sum, our findings were more in line with selection rather than with enrichment effects, with the caveat that our findings rely on work variables taken at baseline. We discuss potential mechanisms underlying these findings.