Effective listening comprehension skills are an important prerequisite for the academic success of primary school students. However, the assessment of listening skills in the instructional language appears to have received only scant attention in the literature. Therefore, the goal of the present study was twofold. Firstly, a comprehensive listening test was developed and different aspects of construct validity supporting the use of the listening test were explored. The listening test was administered to 1001 sixth-grade primary school students in Flanders, the Dutch-speaking part of Belgium. Next, the test items were controlled for item difficulty and discrimination, dimensionality, model-data fit, local item independence, monotonicity, and gender differential item functioning. The final listening test consisted of 24 multiple-choice and open-ended test items. Secondly, the listening test was used to identify differences between students' listening skills based on gender and home language. The results indicate that gender was not significantly related to listening comprehension skills, but L1 Dutch-speaking students significantly outperformed L2 Dutch-speaking students. This study also covers possible further fine-tuning of the instrument.