This study compared the analytic scores on an essay test of clinical judgment generated by nonphysician readers with those of physician readers. Generalizability theory was applied to determine the number of essays and scorers required to reach specified levels of reproducibility. Results suggested that with training nonphysicians could appropriately apply the scoring rules. (MLW)