Visual imagery and mental reconstruction of scenes are considered core components of episodic memory retrieval. Individuals with absent visual imagery (aphantasia) score lower on tests of autobiographical memory, suggesting that aphantasia may be associated with differences in episodic and associative processing. In this online study, we tested aphantasic participants and controls on associative recognition and memory confidence for three types of associations encoded incidentally: associations between visual-visual and audio-visual stimulus pairs, associations between an object and its location on the screen, and intraitem associations. Aphantasic participants had a lower rate of high-confidence hits in all associative memory tests compared with controls. Performance on auditory-visual associations was correlated with individual differences in a measure of object imagery in the aphantasic group but not in controls. No overall group difference in memory performance was found, indicating that visual imagery selectively contributes to memory confidence. Analysis of the encoding task revealed that aphantasics made fewer associative links between the stimuli, suggesting a role for visual imagery in associative processing of visual and auditory input. These data enhance our understanding of visual imagery contributions to associative memory and further characterize the cognitive profile of aphantasia.