Large-scale educational assessments differ from familiar educational measurements by attempting to provide information about the levels and natures of skills in populations rather than in individuals. That the distinct purposes of assessment require different methodologies than individual measurement was recognized by the development of multiple-matrix sampling designs. Similar changes in thinking are necessary when it comes to applying item response theory (IRT) measurement models. This paper outlines three challenges to IRT in the assessment setting, and discusses the responses to them taken in the recent surveys of reading, mathematics, and science undertaken by the National Assessment of Educational Progress. (Author)