This paper reports an investigation into the psychometric properties of the COR-Advantage1.5 (COR-Adv1.5) assessment tool, a criterion-referenced observation-based instrument designed to assess the developmental abilities of children from birth through kindergarten. Using data from 8534 children participating in a state-funded preschool program for disadvantaged 4-year-olds, we employed a multilevel bifactor model to show that CORAdv1.5 has some potential as an unbiased assessment tool for tracking developmental progress in pre-kindergarten children, under a bifactor model parameterization, with fewer dimensions than were proposed by the test developer, HighScope. Examination of factor adequacy indices revealed that, despite the multidimensional structure, CORAdv1.5 items overwhelmingly reflect variance due to the general factors at the child and classroom levels. Evidence for multilevel reliability, response process validity, predictive validity, longitudinal measurement invariance, and generalizability of the items are also reported. The findings of this study have important implications for the use of COR-Adv1.5.