Drawing from the developmental and cognitive mathematics literature, the purpose of this study was to investigate the reliability, validity, and diagnostic utility of a widely-researched number line task in kindergarten. Specifically, the Number Line Assessment 0-100 (NLA 0-100) as compared to an established kindergarten screening measure was examined using (a) regression models and (b) classification accuracy. Five-and six-year-old students (n = 154) were assessed on numeracy measures in the fall and spring of their kindergarten year. The NLA 0-100 had lower predictive validity and lower classification accuracy compared to an early numeracy screening measure. The findings have implications for practice and future research using number line screening assessments. [This paper was published in "Early Childhood Research Quarterly" v55 p119-128 2021.]