NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED615497
Record Type: Non-Journal
Publication Date: 2021
Pages: 9
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Benefits of Alternative Evaluation Methods for Automated Essay Scoring
Andersen, Øistein E.; Yuan, Zheng; Watson, Rebecca; Cheung, Kevin Yet Fong
International Educational Data Mining Society, Paper presented at the International Conference on Educational Data Mining (EDM) (14th, Online, Jun 29-Jul 2, 2021)
Automated essay scoring (AES), where natural language processing is applied to score written text, can underpin educational resources in blended and distance learning. AES performance has typically been reported in terms of correlation coefficients or agreement statistics calculated between a system and an expert human examiner. We describe the benefits of alternative methods to evaluate AES systems and, more importantly, facilitate comparison between AES systems and expert human examiners. We employ these methods, together with "multi-marked" test data labelled by 5 expert human examiners, to guide machine learning model development and selection, resulting in models that outperform expert human examiners. We extend on previous work on a mature feature-based linear ranking perceptron model and also develop a new multitask learning neural network model built on top of a pretrained language model -- DistilBERT. Combining these two models' scores results in further improvements in performance (compared to that of each single model). [For the full proceedings, see ED615472.]
International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A