The test blueprint is important in test development, where it guides the test item writer in creating test items according to the desired objectives and specifications or characteristics (so-called a priori item characteristics), such as the level of item difficulty in the category and the distribution of items based on their difficulty level. Given that the difficulty level of the test items (easy, medium, or hard) created is influenced by the perceptions, knowledge, and experience of the item writer, item analysis based on empirical data using a specific measurement framework needs to be conducted, in addition to evaluation based on expert judgment, to ensure that the test items and the test itself have appropriate characteristics. The present study investigated the extent to which the a priori characteristics (i.e., item difficulty) of the items of the Business English test taken by 4,836 Universitas Terbuka (UT) students aligned with their characteristics when estimated under classical test theory (CTT) and four-parameter logistic (4-PL) IRT models based on empirical data. In light of the two measurement models used, CTT and 4-PL, we extended this study to exploring the comparability of the two models based on the yielded item difficulty and discrimination estimates and the relationship between pseudo-guessing and carelessness parameters. Our study suggested insufficient support for asserting that the characteristics of the items used in the Business English test align with the characteristics expected by the test developers. The exploration of the comparability of the CTT and 4-PL models demonstrated that while the two models were comparable in terms of the item difficulty estimates yielded, they were not comparable for the item discrimination estimates. Our study also did not find a linear association of the pseudo-guessing and carelessness parameters estimated under the 4-PL model. Further findings of our study and their implications, especially on test development practices, are discussed.