Throughout prolonged tasks, visual attention fluctuates temporally in response to the present stimuli, task demands, and changes in available attentional resources. This temporal fluctuation has downstream effects on memory for stimuli presented during the task. Researchers have established that detection of a target (e.g., a square of a color to which participants are instructed to respond with a button press) within a rapid serial visual presentation (RSVP) stream leads to better memory for concurrently presented stimuli than for stimuli presented along with an RSVP distractor (e.g., a square of a color to which participants are instructed to withhold response). Although debates have arisen regarding whether this memory difference, termed the attentional boost effect, results from target-induced enhancement, distractor-induced impairment, or a combination of the two, researchers have largely come to focus on explanations that consider only target-induced memory enhancement. In the present study, we show across three large-sampled experiments a consistent appearance of both target-induced memory enhancement and distractor-induced memory impairment relative to a baseline. In each experiment, participants responded with a spacebar press to squares of one color in an RSVP stream while withholding response to squares of another color and trials with no square (baseline trials). They simultaneously memorized concurrently presented objects. The presence of both enhancement and impairment in these experiments invites the development of new dual-task research that considers distractor-induced memory impairment and the control of temporal selection across tasks.