In contexts where schooling is delivered through a language of instruction (LoI) that is unfamiliar to learners, it has repeatedly been observed that students are reluctant to speak, passive, or even silent. This is commonly attributed to lack of understanding of the language in use in the classroom. Although this is, undoubtedly, an important part of the explanation, this paper presents findings from a thematic analysis of lesson observations, interviews, and ethnographic field-notes from two secondary schools in Tanzania to argue that we should also be paying much closer attention to the socio-emotional environment. Specifically, this paper demonstrates the prevalence of feelings of fear and shame in students' experiences of learning and how these emotions contributed to students' silence. However, it also shows that the impact of these emotions was not the same for all students. Informal classroom 'rules' were found to govern how different groups of students were expected to behave in response to questions from teachers, meaning that the risks of speaking English were greater for some students than others. As such, the prevalence of fear and shame is argued to compound existing inequalities amongst students, thus hindering progress towards equitable, inclusive and safe education for all.