Originally designed to measure reading and passage comprehension in L1 readers, cloze tests continue to be used for L2 assessment purposes. However, there remain disputes about whether or not cloze items can measure beyond local comprehension information, as well as whether or not they are purely a test of reading alone, or if performance can be generalized to broader claims about proficiency. The current study sets out to address both of these issues by drawing on a large pool of cloze items (k = 449) taken from 15 cloze passages that were administered to 675 L1 and 2246 L2 examinees. In conjunction with test scores, a large-scale L1 experiment was conducted using Amazon's Mechanical Turk to determine the level of minimum context required to answer each item. Using Rasch analysis, item function was compared across both groups, with results indicating that cloze items can draw on information at both the sentence and passage level. This seems to suggest further that cloze tests generally tend to measure reading in both L1 and L2 examinees. These findings have important implications for the continued use of cloze tests, particularly in classroom and high-stakes contexts where they are commonly found.