The modality effect refers to the robust finding that memory performance differs for items presented aurally, as compared with visually. Whereas auditory presentation leads to stronger recency performance in immediate recall, visual presentation often produces better primacy performance (the inverse modality effect). To investigate and model these differences, we conducted two large-scale web-based immediate free recall experiments. In both experiments, participants studied visual and auditory word lists of varying lengths and rates of presentation. We observed typical modality and inverse modality effects, while also discovering that participants were more likely to initiate recall from recent items on auditory trials than on visual trials. However, modality effects persisted regardless of the first item recalled. Meanwhile, an analysis of intrusion errors revealed that participants were more likely on visual trials than on auditory trials to erroneously recall words from one list prior. Furthermore, words presented in the same modality as the present list intruded more often than those presented in a different modality. We next developed a retrieved-context account of the modality effect by fitting the Context Maintenance and Retrieval model to data across multiple list lengths. Through our simulations, we demonstrate that the modality effect can be explained by faster contextual drift and stronger context-to-item association formation during auditory presentation, relative to visual. Our modeling shows that modality effects can arise without hypothesizing distinct memory stores for recent and remote information. Finally, we propose that modality effects may derive primarily from the temporal dynamics of stimuli, rather than their modality.