NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED659454
Record Type: Non-Journal
Publication Date: 2023-Sep-27
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Heterogeneous Effects of Game-Based Failure on Student Persistence in an Online Algebra Game
Kirk Vanacore; Adam Sales; Alison Liu; Erin Ottmar
Society for Research on Educational Effectiveness
Background: Persisting after experiencing difficulty allows students to work in the upper ends of their zones of proximal development, where most learning occurs (Ventura et al., 2013; Vygotsky & Cole, 1978). Digital educational games promote productive persistence by allowing students to repeat the same or similar problems until they reach mastery (Owen et al., 2019). One way gamification may influence students' behavior is by encouraging them to persist by re-attempting problems when they have not achieved optimal performance. Gamification has shown mixed effects on student persistence (Gaston & Cooper, 2017; O'Rourke et al., 2016; Chase et al., 2021; Malkiewich et al., 2016). Therefore, the mechanisms that connect different gamification systems to student persistence and learning must be understood to ensure that learning games are developed effectively to support these outcomes. Several studies have found that performance-based rewards -- one type of gamification system -- serve as feedback that may support persistence and learning (Gaston & Cooper, 2017; O'Rourke et al., 2016). Yet, there is a growing appreciation that the effects of behavioral interventions are not homogeneous (Bryan et al., 2021). In this study, we use a regression discontinuity design and log data from an online gamified algebra platform to estimate the heterogeneous effects of a performance-based feedback system on students' persistence behavior. Program: We explore whether different levels of performance-based feedback cause students to persist when struggling in a math game-based online learning program. This program is effective at improving students' understanding of algebra (Chan et al., 2022; Woodrow-Decker et al., 2023) using perceptual learning (Goldstone et al., 2017) and embodied cognition (Abrahamson et al., 2020). Students are asked to transform a starting algebraic expression into a mathematically-equivalent goal state (Figure 1). Each manipulation of the equation counts as a step. Each problem has an optimal solution (i.e. a minimum number of steps to complete the problem). Students receive feedback based on their proximity to the optimal solution (Figure 2). The clovers act as performance contingent reward: three clovers for optimal performance, two clovers if the student was within two steps, and one clover if they were over two steps of optimal. Students are encouraged to replay the problem if their original solution is suboptimal. Research Questions: 1. Do students respond to differences in performance-based rewards by persisting after encountering a challenging problem? 2. Does the effect of the performance-based reward vary based on student characteristics, such as gender, race, participation in English to Speakers of Other Languages (ESOL) classes, and prior knowledge? Setting: The current study utilizes secondary data from a larger efficacy study, which took place during the 2021-22 school year (Decker-Woodrow, In-press). A total of 52 seventh-grade mathematics teachers and their students from 11 middle schools were recruited from a large, suburban district in the Southeastern United States. Although the efficacy study included four conditions, we focus on the abovementioned gamified condition. Participants: The demographics of students randomly assigned to the gamified condition are presented in Table 1. The analysis included 1009 students completing 26,941 attempts of 245 different problems. Design: Since the performance rewards were determined using the number of steps required to complete a problem relative to cut points, the data represent a regression discontinuity design, where the treatment (Z) is receiving a higher performance-based reward (2 clovers vs. 1 clover), and the (discrete) running variable (R) is the students' numbers of steps over optimal. More specifically, we utilize the method established by Sales & Hansen (2020) of limitless regression discontinuity (LRD), which allows for the estimation of an average treatment effect for all subjects in a window around the cutoff, and which is amenable to discrete running variables. LRD is based on the assumption of "residual ignorability;" that the results for control potential outcomes are independent of treatment assignment. Therefore the difference in the average partial residuals generated using the predicted potential outcome assuming no treatment between treatment and control is an estimate of the causal effect. We implement this method using multi-level logistics regressions run using the glmer package in R (Bates et al., 2015), regressing whether the student replayed a problem on their steps over optimal and the cutpoint indicator. The coefficient associated with the cut point is an estimate of the Local Treatment Effect (LATE), and the partial residuals are used to calculate the Limitless Average Treatment Effect (ATE). We included random intercepts for the problems and students. We included random effects of receiving a higher reward to examine the variability of the effects across students. Results: Table 2 presents the regression discontinuity models, and Table 3 presents the LATE and limitless ATE in terms of the probability of replaying the problem. Interactions between the treatment and race/ethnicity or prior algebraic knowledge were non-significant and, therefore, not presented in the table. Overall the treatment of the higher reward decreases the likelihood that students will replay the problem. Notably, the effect is less for male students' experience than for female students and less for ESOL students than those not enrolled in ESOL. Finally, as the effect varied greatly by the student (tau=0.61), we estimated the ATE for each student within their attempted problems (Figure 3). Conclusions: These findings suggest that rewards can have a small but significant effect on students' persistence behaviors in a gamified learning program, but the impacts of the rewards vary greatly. Overall receiving the higher reward demotivated students to replay the problem compared with receiving the lower reward. This suggests that, on average, students are motivated to replay problems by rewards that provide feedback indicating game-like failure. Yet, despite the average effect, students respond differently based on this type of feedback. The smaller effect for ESOL students may be attributed to difficulties understanding the user experience, as the game was presented exclusively in English. The significant difference in feedback effects between males and females is unexpected and should be explored further. Finally, the unexplained heterogeneity of effects (e.g., the variance random effects) suggests the necessity of future research on who is influenced by the gamified elements of this program.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: Elementary Education; Grade 7; Junior High Schools; Middle Schools; Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A