NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
ERIC Number: ED661626
Record Type: Non-Journal
Publication Date: 2024
Pages: 153
Abstractor: As Provided
ISBN: 979-8-3840-4015-6
ISSN: N/A
EISSN: N/A
Developing a Generic Scorer for Practice Writing Tests of Statewide Assessment Essays with Natural Language Processing Transfer Learning Techniques
Yi Gui
ProQuest LLC, Ph.D. Dissertation, The University of Iowa
This study explores using transfer learning in machine learning for natural language processing (NLP) to create generic automated essay scoring (AES) models, providing instant online scoring for statewide writing assessments in K-12 education. The goal is to develop an instant online scorer that is generalizable to any prompt, addressing the current limitations of online writing practice tests for operational assessments, such as those for statewide writing assessment (SWAS). The study leverages Google's BERT, a state-of-the-art NLP transfer learning AI product, to train and build generic essay scoring models which are based on statistical ground of ordinal logistic regression (OLR) via machine learning. Three groups were analyzed: a control group with no additional pre-training, a group further pre-trained on 12,970 ASAP essays (in-domain materials), and a group further pre-trained on 500 SWAS essays (within-task materials). Models were trained with 9th- and 11th-grade SWAS essays and evaluated on 10th-grade essays. Model evaluation metrics included Quadratic Weighted Kappa (QWK), Mean Absolute Errors (MAEs), accuracy, precision, recall, and F1-score. Results indicated that further pre-training does not necessarily enhance scoring performance. The control group often matched or exceeded the SWAS pre-trained group. In all three groups, the scoring patterns of Language Use are consistent, while the ASAP pre-trained group excelled in scoring the Prompt Task trait. These findings highlight the importance of the pre-training materials' quality and indicate that further pre-training does not necessarily improve model performance for distinct downstream tasks. Future research should use balanced datasets with more essay prompts and explore different experimental designs to find the prerequisites of further pre-training to improve the AES model performance. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://bibliotheek.ehb.be:2222/en-US/products/dissertations/individuals.shtml.]
ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://bibliotheek.ehb.be:2222/en-US/products/dissertations/individuals.shtml
Publication Type: Dissertations/Theses - Doctoral Dissertations
Education Level: Early Childhood Education; Elementary Education; Kindergarten; Primary Education; Elementary Secondary Education; Grade 9; High Schools; Junior High Schools; Middle Schools; Secondary Education; Grade 10; Grade 11
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A