English as a medium of instruction and communication is becoming a central pedagogy in various countries in the world. In Pakistan, most of the advanced academic institutions use English as their medium of instruction, however students and teachers have been observed communicating in their first languages, especially Urdu, in the classrooms. This study was conducted at the Institute of Communication and Cultural Studies, University of Management and Technology, to examine the factors that limit students and teachers from communicating in English. Qualitative research methods were employed to find the factors affecting the use of English in English-only classrooms. A total of 130 participants were included in this study. Among them, 68 participants were engaged in student questionnaires. The participants were students of four native languages (Urdu, Punjabi, Siraiki and Pushto) with different level of English language proficiency. Required literature search was performed using key databases and journals, related to applied linguistics, English language and education. Our results demonstrated that most of the students like to communicate in English and be a part of the English-only environment, however they acknowledged some key factors that prevented them from speaking English in the classrooms. Fundamental factors included linguistics, individual, social, psychological and institutional which were largely based upon their beliefs, assumptions, needs and desires. This study may provide insights to the policymakers to work on issues related to communications and medium of instructions in Pakistan. It may also help teachers and educators to understand the underlying factors that compel students to speak their first languages instead of English.