Second Language (L2) confidence is considered an affective variable for L2 users to claim ownership of English. However, the findings of previous studies could not be generalized to international students in a non-English-speaking context like Taiwan since English mainly functions as a lingua franca among individuals with diverse first languages. This study investigated three international students' development of L2 confidence while they were studying in Taiwanese universities. Qualitative semi-structured interviews were conducted on personal background, experiences, critical events, and personal evaluation of their L2 confidence. The interviews were transcribed, coded, and then analyzed in thematic narratives. Specific situations that affected this dynamic L2 confidence development were identified and probed deeper. The findings showed that the three participants seemed more confident in an English as Lingua Franca (ELF) community where local students were absent. Native Speaker (NS)-norms still dominated their English journey, and their confidence level greatly hinged on their NS-based proficiency. The ideology of following the NS English model threatened L2 confidence in ELF interactions. Furthermore, the participants' perception of their nonnative speaker (NNS) identity might make them less confident. The findings contribute to the pedagogical implications for L2 learners, users and teachers.