ERIC Number: ED624071
Record Type: Non-Journal
Publication Date: 2022
Pages: 11
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Individual Fairness Evaluation for Automated Essay Scoring System
Doewes, Afrizal; Saxena, Akrati; Pei, Yulong; Pechenizkiy, Mykola
International Educational Data Mining Society, Paper presented at the International Conference on Educational Data Mining (EDM) (15th, Durham, United Kingdom, Jul 24-27, 2022)
In Automated Essay Scoring (AES) systems, many previous works have studied group fairness using the demographic features of essay writers. However, individual fairness also plays an important role in fair evaluation and has not been yet explored. Initialized by Dwork et al., the fundamental concept of individual fairness is "similar people should get similar treatment". In the context of AES, individual fairness means that "similar essays should be treated similarly". In this work, we propose a methodology to measure individual fairness in AES. The similarity of essays can be computed using the distance of the text representation of essays. We compare several text representations of essays, from the classical text features, such as BOW and TFIDF, to the more recent deep-learning-based features, such as Sentence-BERT and LASER. We also show their performance against paraphrased essays to understand if they can maintain the ranking of similarities between the original and the paraphrased essays. Finally, we demonstrate how to evaluate the performance of automated scoring systems models with regard to individual fairness by counting the number of pairs of essays that satisfy the individual fairness equation and by observing the correlation of score difference with the distance of essays. Our analysis suggests that the Sentence-BERT, as the text representation of the essays, and Gradient Boosting, as the score prediction model, provide better results based on the proposed individual fairness evaluation methodology. [For the full proceedings, see ED623995.]
Descriptors: Scoring, Essays, Writing Evaluation, Comparative Analysis, Computer Assisted Testing, Ethics, Student Evaluation, Computational Linguistics, Text Structure, Learning Processes, Phrase Structure, Prediction, Models, Word Frequency
International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A