ERIC Number: EJ1343353
Record Type: Journal
Publication Date: 2022
Pages: 19
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: EISSN-2229-0443
Raters' Perceptions of Rating Scales Criteria and Its Effect on the Process and Outcome of Their Rating
Heidari, Nasim; Ghanbari, Nasim; Abbasi, Abbas
Language Testing in Asia, v12 Article 20 2022
It is widely believed that human rating performance is influenced by an array of different factors. Among these, rater-related variables such as experience, language background, perceptions, and attitudes have been mentioned. One of the important rater-related factors is the way the raters interact with the rating scales. In particular, how raters perceive the components of the scales to further plan their scoring seems important. For this aim, the present study investigated the raters' perceptions of the rating scales and their subsequent rating behaviors for two analytic and holistic rating scales. Hence, nine highly experienced raters were asked to verbalize their thoughts while rating student essays using IELTS holistic scale and the analytic scale of ESL Composition Profile. Upon analyzing the think-aloud protocols, four themes emerged. The findings showed that when rating holistically, the raters either referred to the holistic scale components to validate their ratings (validation) or had a pre-evaluation reading to rate in a more reliable way (dominancy). In analytic rating, on the other hand, the raters used a pre-evaluation scale reading in order to keep the components and their criteria to memory to evaluate the text more accurately (dominancy) or regularly moved between the text and the scale components to assign a score (oscillation). Furthermore, the results of a Wilcoxon signed-rank test showed that when using the holistic and analytic rating scales, the raters assigned significantly different scores to the texts. On the whole, the results revealed that the way the raters perceived the scale components will affect their judgement of the texts. The study also provides several implications for rater training programs and EFL writing assessment.
Descriptors: Evaluators, Rating Scales, Language Tests, English (Second Language), Second Language Learning, Essays, Writing Evaluation, Second Language Instruction, Profiles, Protocol Analysis, Scoring, Holistic Approach, Decision Making, Training, Native Language, Attitudes, Evaluation Criteria
Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://bibliotheek.ehb.be:2123/
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Assessments and Surveys: International English Language Testing System
Grant or Contract Numbers: N/A