Setting standards for the purpose of certification is frequently performed using judgmental techniques such as the Angoff method. This study was performed to identify types of items that judges find hard to rate accurately, that is, types of items on which examinees perform differently than predicted by the judges. Once identified these item types should be the subject of additional training for future panels of judges. Observed difficulty estimates were regressed on the Angoff estimates of a panel of judges for three examinations and items with large residuals were identified. For one examination there were indications that the judges found negatively phrased items hard to rate accurately. For a second examination there were indications that the judges found items involving calculations as hard to rate accurately. In both cases the judges tended to underestimate the difficulty of the items; the examinees found the items harder than the judges predicted. No discernible pattern was found for the third examination. (Author)