Fostering the listening comprehension of English as Foreign Language (EFL) learners has been recognized as an important and challenging issue. Videos have been used as one of the English listening learning resources; however, without effective learning supports, EFL students are likely to encounter difficulties in comprehending the video content, leading to frustration and reducing learning interest. In this study, a learning system with an automatic caption filtering and partial hiding mechanism was developed to improve the English listening comprehension of EFL students. An experiment was conducted to evaluate the effects of the proposed approach on students' learning achievements and perceptions. The participants were 76 freshmen from two classes of a non-language-related department of a university. Each class contained 38 students. The two classes of students were situated in the proposed learning context and the conventional technology-enhanced learning context with a counterbalanced experimental design. The experimental results verified that the proposed caption filtering approach effectively improved the listening comprehension of the students in comparison with the conventional approach which provides full captions and an e-dictionary. Moreover, from the data collected by the eye movement tracking device, it was found that more than 76% of the students relied on the captions during the learning activity, reflecting the importance of developing effective caption filtering mechanisms for supporting EFL learners.