This paper describes procedures for estimating various indices of classification consistency and accuracy for multiple category classifications using data from a single test administration. The estimates of the classification consistency and accuracy indices are compared under three different psychometric models: the two-parameter beta binomial, four-parameter beta binomial, and three-parameter logistic item response theory (IRT) models. Using real data sets, the estimation procedures are illustrated, and the characteristics of the estimated indices are examined. This paper also examines the behavior of the estimated indices as a function of the latent variable. The IRT model tends to provide better fits to the data used in this study, and shows larger estimated consistency and accuracy. Although the results are not substantially different across different models, all three components of the models (i.e., the estimated true score distributions, fitted observed score distributions, and estimated conditional error variances) appear to have a great influence on estimates of the indices. Choosing a model in practice should be based on various considerations including the degree of the model fit to the data, suitability of the model assumptions, and the computational feasibility. (Contains 7 tables, 16 figures, and 37 references.) (Author/SLD)