Automated Essay Scoring (AES) tools aim to improve the efficiency and consistency of essay scoring by using machine learning algorithms. In the existing research work on this topic, most researchers agree that human-automated score agreement remains the benchmark for assessing the accuracy of machine-generated scores. To measure the performance of…

Scoring essays is generally an exhausting and time-consuming task for teachers. Automated Essay Scoring (AES) facilitates the scoring process to be faster and more consistent. The most logical way to assess the performance of an automated scorer is by measuring the score agreement with the human raters. However, we provide empirical evidence that…

The main purpose of the study is to evaluate the qualities of human essay ratings for a large-scale assessment using Rasch measurement theory. Specifically, Many-Facet Rasch Measurement (MFRM) was utilized to examine the rating scale category structure and provide important information about interpretations of ratings in the large-scale…

This paper explores the annotation and classification of students' revision behaviors in argumentative writing. A sentence-level revision schema is proposed to capture why and how students make revisions. Based on the proposed schema, a small corpus of student essays and revisions was annotated. Studies show that manual annotation is reliable with…

We investigated linguistic factors that relate to misalignment between students' and teachers' ratings of essay quality. Students (n = 126) wrote essays and rated the quality of their work. Teachers then provided their own ratings of the essays. Results revealed that students who were less accurate in their self-assessments produced essays that…

This study examined the reliability of scores assigned to the essays written by Kentucky students to meet the University Writing Requirement (UWR) at Eastern Kentucky University. Two sets of essays, 50 each, on the same prompt that had been read and scored in 1989 and 1997 by trained UWR scorers were read by 7 UWR scorers in 2000. A correlation…

This paper reports the results of an analysis of the relationship between scorer behaviors and score variability. Thirty-six essay scorers were interviewed and asked to perform a think-aloud task as they scored 24 essays. Each comment made by a scorer was coded according to its content focus (i.e. appearance, assignment, mechanics, communication,…

The faculty of an intensive program of English as a second language for college-bound students, questioning the objectivity of faculty evaluations of non-native college applicants' written essays, assessed the existing evaluation process, reformed it, tested it, and planned for ongoing development. In the first stage, readers read and graded…

Contradictions are inherent in the evaluation of placement test writing, contradictions that at once value and devalue writers and writing, readers and reading. In testing, the evidence for the essay's effectiveness rests almost entirely on the writer's choice of linguistic forms. The characteristics that distinguish evaluation in competency…

Over 2,000 writing samples were collected from four undergraduate institutions and compared, where possible, with scores on a multiple-choice test. High correlations between ratings of the writing samples and multiple-choice test scores were obtained. Samples contributed substantially to the prediction of both college grades and writing…

This study examined the difficulty level of essay topics used in the large-scale assessment of writing in relation to five different scoring models, and sought to determine what effects the scoring models would have on passing rates. In model one, examinee's score is the direct result of a score assigned by the reader or the sum of scores assigned…

A chronicle of the problems faced in an attempt to collect data on sociology curriculum trends in California's community college system is presented. The project was initiated in an effort to determine if other colleges in the system were experiencing the same difficulties with curriculum and enrollment in sociology courses being encountered by…

The validity and reliability of essay scores were examined by comparing the mean scores assigned to good and poor quality essay responses of different lengths written by high school sophomores. In-service and pre-service social studies teachers graded essay responses to a test question requiring knowledge of the Constitutional provisions for…

The relationship among writing assessment strategies which may be applicable to competency based testing was examined in a study involving 128 high school students in six English classes in grades 11 and 12. Each student wrote two essays of at least 200 words on topics designed to test expository, or explanatory writing, and completed an objective…

Four systems for scoring student essays were used to classify eleventh grade and undergraduate students according to writing ability. The reliabilities of the raters and the validities of the systems in classifying students were emphasized. Two analytic scoring systems--which assume that quality writing is characterized by the inclusion of certain…