ERIC Number: EJ1353541
Record Type: Journal
Publication Date: 2022-Dec
Pages: 23
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0922-4777
EISSN: EISSN-1573-0905
Scoring Rubric Reliability and Internal Validity in Rater-Mediated EFL Writing Assessment: Insights from Many-Facet Rasch Measurement
Reading and Writing: An Interdisciplinary Journal, v35 n10 p2409-2431 Dec 2022
Scoring rubrics are known to be effective for assessing writing for both testing and classroom teaching purposes. How raters interpret the descriptors in a rubric can significantly impact the subsequent final score, and further, the descriptors may also color a rater's judgment of a student's writing quality. Little is known, however, about how peer raters use a teacher-developed scoring rubric in English as a Foreign Language (EFL) writing contexts. In the current study, Many-Facet Rasch Measurement (MFRM) was applied to examine a scoring rubric for EFL writing and analyze the severity and consistency in rating behaviors between teachers and peer raters. The findings revealed four key points: (1) students' writing skills can be differentiated by this scoring rubric which measured the construct as expected; (2) four rating criteria were reasonably designed, but the scoring bands were not wide enough to separate students' writing ability; (3) teachers were stricter raters than student peers, but both showed a central tendency effect; and (4) teachers had outstanding intra-rater reliability, while peers showed higher inter-rater reliability. The findings provide some evidence for the reliability and internal validity of the scoring rubric and indicate that teachers could use it for assessing EFL writing; however, the scoring bands would need to be broadened to be applied to a broader range of students' writing levels. Implications for introducing scoring rubrics in peer-mediated assessment for teaching writing, developing scoring rubrics in EFL writing assessment, and using MFRM to evaluate scoring rubrics are discussed.
Descriptors: Scoring Rubrics, Interrater Reliability, Writing Evaluation, Teaching Methods, English (Second Language), Second Language Learning, Second Language Instruction, Writing Skills, Evaluators, Peer Evaluation, Evaluation Criteria, Language Teachers, Reliability
Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://bibliotheek.ehb.be:2123/
Publication Type: Journal Articles; Reports - Research; Tests/Questionnaires
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A