Visual working memory (VWM) is an online memory buffer that is typically assumed to be immune to source memory confusions. Accordingly, the few studies that have investigated the role of proactive interference (PI) in VWM tasks found only a modest PI effect at best. In contrast, a recent study has found a substantial PI effect in that performance in a VWM task was markedly improved when all memory items were unique compared to the more standard condition in which only a limited set of objects was used. The goal of the present study was to reconcile this discrepancy between the findings, and to scrutinize the extent to which PI is involved in VWM tasks. Experiments 1-2 showed that the robust advantage in using unique memory items can also be found in a within-subject design and is largely independent of set size, encoding duration, or intertrial interval. Importantly, however, PI was found mainly when all items were presented at the same location, and the effect was greatly diminished when the items were presented, either simultaneously (Experiment 3) or sequentially (Experiments 4-5), at distinct locations. These results indicate that PI is spatially specific and that without the assistance of spatial information VWM is not protected from PI. Thus, these findings imply that spatial information plays a key role in VWM, and underscore the notion that VWM is more vulnerable to interference than is typically assumed.