NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
PDF on ERIC Download full text
ERIC Number: ED630844
Record Type: Non-Journal
Publication Date: 2023
Pages: 7
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Effective Evaluation of Online Learning Interventions with Surrogate Measures
Prihar, Ethan; Vanacore, Kirk; Sales, Adam; Heffernan, Neil
International Educational Data Mining Society, Paper presented at the International Conference on Educational Data Mining (EDM) (16th, Bengaluru, India, Jul 11-14, 2023)
There is a growing need to empirically evaluate the quality of online instructional interventions at scale. In response, some online learning platforms have begun to implement rapid A/B testing of instructional interventions. In these scenarios, students participate in series of randomized experiments that evaluate problem-level interventions in quick succession, which makes it difficult to discern the effect of any particular intervention on their learning. Therefore, distal measures of learning such as posttests may not provide a clear understanding of which interventions are effective, which can lead to slow adoption of new instructional methods. To help discern the effectiveness of instructional interventions, this work uses data from 26,060 clickstream sequences of students across 31 different online educational experiments exploring 51 different research questions and the students' posttest scores to create and analyze different proximal surrogate measures of learning that can be used at the problem level. Through feature engineering and deep learning approaches, next-problem correctness was determined to be the best surrogate measure. As more data from online educational experiments are collected, model based surrogate measures can be improved, but for now, next-problem correctness is an empirically effective proximal surrogate measure of learning for analyzing rapid problem-level experiments. [For the complete proceedings, see ED630829. Additional funding for this paper was provided by the U.S. Department of Education's Graduate Assistance in Areas of National Need (GAANN).]
International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: National Science Foundation (NSF); Institute of Education Sciences (ED); Office of Elementary and Secondary Education (OESE) (ED), Education Innovation and Research (EIR); Office of Naval Research (ONR) (DOD); Federal Highway Administration (FHWA), National Highway Institute (NHI)
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: 2118725; 2118904; 1950683; 1917808; 1931523; 1940236; 1917713; 1903304; 1822830; 1759229; 1724889; 1636782; 1535428; R305N210049; R305D210031; R305A170137; R305A170243; R305A180401; R305A120125; U411B190024; S411B210024; N000141812768; R44GM146483
Data File: URL: https://osf.io/uj48v/