Abstract:
In this article, we investigate a systematic workflow that supports the learning engineering process of formulating the starting question for a conversational module base...Show MoreMetadata
Abstract:
In this article, we investigate a systematic workflow that supports the learning engineering process of formulating the starting question for a conversational module based on existing learning materials, specifying the input that transformer-based language models need to function as classifiers, and specifying the adaptive dialogue structure, i.e., the turns the classifiers can choose between. Our primary purpose is to evaluate the effectiveness of conversational modules if a learning engineer follows our workflow. Notably, our workflow is technically lightweight, in the sense that no further training of the models is expected. To evaluate the workflow, we created three different conversational modules. For each, we assessed classifier quality and how coherent the follow-up question asked by the agent was based on the classification results of the user response. The classifiers reached F1-macro scores between 0.66 and 0.86, and the percentage of coherent follow-up questions asked by the agent was between 79% and 84%. These results highlight, first, the potential of transformer-based models to support learning engineers in developing dedicated conversational agents. Second, it highlights the necessity to consider the quality of the adaptation mechanism together with the adaptive dialogue. As such models continue to be improved, their benefits for learning engineering will rise. Future work would be valuable to investigate the usability of this workflow by learning engineers with different backgrounds and prior knowledge on the technical and pedagogical aspects of learning engineering.
Published in: IEEE Transactions on Learning Technologies ( Volume: 17)
Funding Agency:

Behzad Mirzababaei received the M.A. degree in computer science from the University of Tehran, Tehran, Iran, in 2014. He is currently working toward the Ph.D. degree in computer science with the Graz University of Technology, Graz, Austria.
He is currently a Researcher with the Know-Center, a not-for-profit research company. His research interests include technology-enhanced learning, argument mining, tutoring systems, and...Show More
Behzad Mirzababaei received the M.A. degree in computer science from the University of Tehran, Tehran, Iran, in 2014. He is currently working toward the Ph.D. degree in computer science with the Graz University of Technology, Graz, Austria.
He is currently a Researcher with the Know-Center, a not-for-profit research company. His research interests include technology-enhanced learning, argument mining, tutoring systems, and...View more

Know-Center GmbH, Graz, Austria
Graz University of Technology, Graz, Austria
Viktoria Pammer-Schindler received the doctoral and Habilitation degrees in computer science from the Graz University of Technology, Graz, Austria, in 2010 and 2019, respectively.
Since 2012, she has been a Research Area Head at the Know-Center, a not-for-profit research company. Since 2020, she has been an Associate Professor at TU Graz, Graz. Her research interests include socio-technical interventions for learning and k...Show More
Viktoria Pammer-Schindler received the doctoral and Habilitation degrees in computer science from the Graz University of Technology, Graz, Austria, in 2010 and 2019, respectively.
Since 2012, she has been a Research Area Head at the Know-Center, a not-for-profit research company. Since 2020, she has been an Associate Professor at TU Graz, Graz. Her research interests include socio-technical interventions for learning and k...View more

Behzad Mirzababaei received the M.A. degree in computer science from the University of Tehran, Tehran, Iran, in 2014. He is currently working toward the Ph.D. degree in computer science with the Graz University of Technology, Graz, Austria.
He is currently a Researcher with the Know-Center, a not-for-profit research company. His research interests include technology-enhanced learning, argument mining, tutoring systems, and educational chatbots.
Behzad Mirzababaei received the M.A. degree in computer science from the University of Tehran, Tehran, Iran, in 2014. He is currently working toward the Ph.D. degree in computer science with the Graz University of Technology, Graz, Austria.
He is currently a Researcher with the Know-Center, a not-for-profit research company. His research interests include technology-enhanced learning, argument mining, tutoring systems, and educational chatbots.View more

Know-Center GmbH, Graz, Austria
Graz University of Technology, Graz, Austria
Viktoria Pammer-Schindler received the doctoral and Habilitation degrees in computer science from the Graz University of Technology, Graz, Austria, in 2010 and 2019, respectively.
Since 2012, she has been a Research Area Head at the Know-Center, a not-for-profit research company. Since 2020, she has been an Associate Professor at TU Graz, Graz. Her research interests include socio-technical interventions for learning and knowledge work, and the intersection of technology-enhanced learning and human–computer interaction.
Dr. Pammer-Schindler was the President of the International Alliance to Advance Learning in the Digital Era, a Member of the European Association of Technology-Enhanced Learning's Managing Committee, and a Subcommittee Chair with the ACM Conference on human factors in computing.
Viktoria Pammer-Schindler received the doctoral and Habilitation degrees in computer science from the Graz University of Technology, Graz, Austria, in 2010 and 2019, respectively.
Since 2012, she has been a Research Area Head at the Know-Center, a not-for-profit research company. Since 2020, she has been an Associate Professor at TU Graz, Graz. Her research interests include socio-technical interventions for learning and knowledge work, and the intersection of technology-enhanced learning and human–computer interaction.
Dr. Pammer-Schindler was the President of the International Alliance to Advance Learning in the Digital Era, a Member of the European Association of Technology-Enhanced Learning's Managing Committee, and a Subcommittee Chair with the ACM Conference on human factors in computing.View more