Memory representations can be stored in a passive state in a visual working memory (VWM) task. However, it remains unclear whether the representations stored in the passive state are prone to interference and decay. To explore this issue, we asked participants to successively remember two sets of memory items (M1 and M2) in three test manners: a "combined test" (both M1 and M2 are probed simultaneously), a "backward test" (probe M2 first and M1 second), or a "forward test" (probe M1 first and M2 second). We found that the contralateral delay activity (CDA) amplitude after the onset of M2 only tracked M2 independently of M1 in the two "separate tests" (Experiments 1-3), and the accuracy of M1 was well above chance. These results implied that the M1 representations had been transferred from the online state into the passive state after the onset of M2. Furthermore, the accuracy of M1 (two representations were transferred from the online state into the passive state and retrieved later) in the "backward test" was worse than M2 (2 representations in the online state throughout) in the "backward test" (Experiments 1-2), but was comparable to M1 (two representations were transferred from the online state into the passive state and retrieved first) in the "forward test" (Experiment 2). These results demonstrated that the memory representations were impaired during state switching. Importantly, once the representations had been stored in the passive state, they were robust with little memory loss during latent retention.