Recall performance in working memory (WM) is strongly affected by the similarity between items. When asked to encode and recall list of items in their serial order, people confuse more often the position of similar compared to dissimilar items. Models of WM explain this deleterious effect of similarity through a problem of discriminability between WM representations. In contrast, when lists of items that are all semantically similar are compared to lists of dissimilar items, semantic similarity does not negatively impact order memory, questioning the idea that semantic information is part of the WM content. This study reports four experiments in which semantic similarity was manipulated using lists composed of multiple semantic categories. These experiments revealed two main patterns. First, semantic similarity can "increase", rather than decrease, order memory. Second, semantic knowledge reliably constrains the way items migrate; when migrating, items tend to do so more often toward the position of other similar items, rather than migrating toward other dissimilar items. These results challenge the way current models of WM represent similarity. The plausibility of different theoretical accounts and mechanisms is discussed.