Using Chinese characters, we investigated how stroke count and frequency of use influence attention and short-term memory (STM) encoding in Mainland Chinese speakers. To isolate specific components of attention we employed the Theory of Visual Attention (TVA), which allowed estimates of STM capacity, processing speed, and the threshold of visual perception. An analysis of TVA parameters revealed that familiarity affects both the memory capacity and processing speed of objects, whereas the threshold for visual perception remained unaffected. Interestingly, our results also indicate that modulation of attention is driven solely by familiarity with the characters, independent of the actual physical aspect of Chinese characters. We propose that mental categories and prior knowledge play a vital role in the processing of information in attention, as well as in how this information is stored and represented in visual STM.