This paper reports the results of an analysis of the relationship between scorer behaviors and score variability. Thirty-six essay scorers were interviewed and asked to perform a think-aloud task as they scored 24 essays. Each comment made by a scorer was coded according to its content focus (i.e. appearance, assignment, mechanics, communication, organization, story, or style) and its processing action (i.e. diagnose, monitor, review, or rationale). The number of comments made by each scorer that fell into each of these categories was regressed onto the variability of the scores assigned to each of the 24 essays. The results show that direct writing assessment scores are less reliable on essays for which scorers focus on abstract and difficult-to-define features. More specifically, these scorers showed less agreement for essays when their evaluative comments focused on the way the story is communicated by the writer and the way that the writer's sentence structure, vocabulary, and voice convey an individual's writing style. Scorers were also more likely to use less efficient scoring strategies for essays such as these. That is, they were more likely to break the scoring task down into subtasks by using a monitoring strategy. They were also more likely to diagnose ways that the writing could be improved. (Author)