This study aimed to compare item characteristics and response time between stimulus conditions in computer-delivered listening tests. Listening materials had three variants: regular videos, frame-by-frame videos, and only audios without visuals. Participants were 228 Japanese high school students who were requested to complete one of nine listening test forms. Results showed that item difficulty did not significantly differ between conditions, but item discrimination was much weaker in regular video versions than in the other two conditions. Response time was not different in the three conditions in any quantile. This suggests that listening items full of visual information have different aspects of the general (traditional) listening construct, despite the possibility of construct-irrelevant factors being affected.