This study identifies challenges perceived by Indonesian students of English as a Foreign Language in "Oracy in Academic Context" classes as well as possible causes of the problems. This subject was a new addition to the 2018 curriculum of the English Department of Teacher Training and Education Faculty at XXX University in Indonesia. The findings show the main problems students faced related to speaking issues, for example, lack of self-confidence, fear of making mistakes, ineffective class schedule, annoying audience behaviour, lack of vocabulary, lack of grammatical precision, poor pronunciation, difficult and uninteresting topics, direct or immediate corrective feedback, and problems coordinating group work. The students also pointed out that as listeners they perceived difficulties due to their lack of adequate background knowledge, unfamiliar vocabularies, anxiety, speakers' poor pronunciation, too fast or too long presentations, accents, clarity of the sound, seat positioning, room temperature, lack of concentration, and indifferent speakers. Based on analysis of the data, these problems are induced by several factors which include lack of linguistic proficiency, variable cognitive competencies, physical conditions, and most prominently psychological, social and emotional challenges. A set of recommendations to address these issues is also presented in the following paper.