Visual working memory (VWM) refers to the temporary storage and manipulation of visual information. Although visually different, objects we view and remember can share the same higher-level category information, such as an apple, orange, and banana all being classified as fruit. We study the influence of category information on VWM, focusing on the question of whether stimulus category coherence (i.e., whether all to-be-remembered items belong to the same semantic category) influences VWM performance. This question is addressed in two behavioral experiments using a change-detection paradigm and a rational analysis using an ideal observer based on a Bayesian model. Both experimental participants and the ideal observer often, but not always, performed numerically better on coherent trials (i.e., when all stimuli belonged to the same category). We hypothesize that the influence of category coherence information on VWM may be task-dependent and/or stimulus-dependent. In conditions when category coherence information is highly valuable for task performance, as indicated by the ideal observer, then participants tended to make use of it. However, when the ideal observer suggested this information was not crucial to performance, participants did not. In addition, both participants and the ideal observer showed a bias toward responding "same," and often showed a stronger influence of category coherence on change trials. The consistencies between participant and ideal observer responses suggest participants often behaved as they did because these behaviors are optimal (or approximately so) for maximizing task performance. This may help explain conflicting results reported in the scientific literature.