Instructional Manipulation Checks (IMCs) are intended to detect inattention, a common occurrence in survey responding. We use eye tracking to empirically assess the attention that survey respondents dedicate to a short and a long IMC. We find that all 21 respondents pass the short IMC. In contrast, six respondents fail the long IMC. Our eye-tracking results show that some respondents who fail the long IMC check allocate more cognitive effort and time to processing this check than others, suggesting that lack of comprehension rather than inattention may be the relevant underlying mechanism for their failure. For other respondents failing the IMC, lack of attention and speeding is more likely to be the culprit. These findings culminate into the suggestion that future researchers should assess the time that respondents failing an IMC dedicated to this check to distinguish between miscomprehension (long time) and inattention (short time).