The global outbreak of the new coronavirus epidemic has promoted the development of intelligent education and the utilization of online learning systems. In order to provide students with intelligent services, such as cognitive diagnosis and personalized exercises recommendation, a fundamental task is the concept tagging for exercises, which extracts knowledge index structures and knowledge representations for exercises. Unfortunately, to the best of our knowledge, existing tagging approaches based on exercise content either ignore multiple components of exercises or ignore that exercises may contain multiple concepts. To this end, in this article, we present a study of concept tagging. First, we propose an improved pretrained bidirectional encoder representations from transformers (BERT) for concept tagging with both questions and solutions (QSCT). Specifically, we design a question-solution prediction task and apply the BERT encoder to combine questions and solutions, ultimately obtaining the final exercise representation through feature augmentation. Then, to further explore the relationship between questions and solutions, we extend the QSCT to a pseudo-siamese BERT for concept tagging with both questions and solutions. We optimize the feature fusion strategy, which integrates five different vector features from local and global into the final exercise representation. Finally, we conduct extensive experiments on real-world datasets, which clearly demonstrate the effectiveness of our proposed models for concept tagging.