We introduce a cognitive framework for measuring reading comprehension that includes the use of novel summary-writing tasks. We derive NLP features from the holistic rubric used to score the summaries written by students for such tasks and use them to design a preliminary, automated scoring system. Our results show that the automated approach…

This study explores correlations between human ratings of essay quality and component scores based on similar natural language processing indices and weighted through a principal component analysis. The results demonstrate that such component scores show small to large effects with human ratings and thus may be suitable to providing both summative…

A study investigated two questions regarding the validity of holistic writing evaluation: (1) How well do holistic scores predict sentence length, passage length, and spelling errors in grades 3, 6, and 9 narrative and explanatory writing? and (2) Which model best predicts these variables (across grades within type, or within grade within type)?…

In 1989 the Portland (Oregon) Public Schools conducted a Direct Writing Assessment of approximately 6,000 students. A voluntary portion included 103 ninth graders who wrote essays on the same topic. The papers were scored twice: once in each student's handwriting and once in word-processed form. Papers were separated into short and long categories…

The use of confirmatory factor analytic procedures to examine the dimensionality of writing skills as measured by a large-scale direct writing test was illustrated. Internal construct validity evidence about the nature of writing skills measured by the test was provided. Data used were scores assigned by about 100 trained professional raters on a…

The viability of the Partial Credit Model (PCM) of G. N. Masters (1982) for scaling and banking individual essay prompts from a statewide writing assessment program was investigated. Preliminary analyses indicated the feasibility of using the PCM, the general polychotomous form of the Rasch model, with essay data from the Maryland Writing Test…

A procedure was developed to detect differential item functioning (DIF) in a standardized essay test using the Partial Credit Model, the general polychotomous form of the Rasch model. Using a panel of experts in the writing process, hypothesized explanations for DIF at some score points were developed. Data for the study included averaged…

The current New Jersey high school graduation law requires that students pass a ninth grade test in reading, writing, and mathematics. Equating the writing test from year to year to be sure that passing scores remain identical, regardless of the test form administered, requires developing a scoring technique. A research study was developed based…

The administration of the High School Proficiency Test (HSPT) in the Trenton (New Jersey) School District and the impact of a voluntary summer tutorial program on HSPT scores were examined. New Jersey requires that all ninth graders take the HSPT, and that all school districts achieve a 75 percent passing rate to become state-certified. The same…

In 1991 Pennsylvania began implementation of a direct writing assessment at the sixth-grade and ninth-grade levels. A total of 18,758 sixth graders and 16,575 ninth graders wrote a response to 1 of 9 prompts reflecting 3 modes of writing. A six-point holistic scale was used to score the papers, with two readers scoring each paper. A third reader,…

This paper reports on the New Jersey State Department of Education's research process for developing a scoring rubric that assesses test objectives and higher order thinking skills through writing. New Jersey's statewide assessment program includes a written essay portion to test students' ability to write on a given topic and communicate an…