The prime objectives of this study were to unveil the level of anxiety and the sources of English language anxiety of the tertiary level students of Bangladesh. Through a questionnaire survey using Foreign Language Classroom Anxiety Scale developed by Horwitz et al. (1986), the study found that most of the tertiary level learners have high anxiety of English as a foreign language. Based on an exploratory factor analysis with principal component analysis and varimax rotation (Kaiser Normalization), a four-factor (communication anxiety, comprehension and test anxiety, fear of negative evaluation, and lack of anxiety) solution emerged. Following a close analysis of the EFA and other established valid models, the study found that six components, namely, speaking anxiety, fear of negative evaluation, peer pressure/social-image anxiety, comprehension-related anxiety, negative attitude towards class, and test anxiety were the main sources of anxiety, which need to be addressed for alleviating the anxiety of the students.