Game-based learning has good chemistry with chemistry education: A three-level meta-analysis

1.1 Learning outcomes in chemistry game-based learning

Chemistry learning involves not only scientific practices (e.g., ask questions; develop and use models; plan and carry out investigations), crosscutting concepts that bridge across other disciplines (e.g., patterns; cause and effect; scale, proportion, and quantity), and chemistry core ideas (e.g., matter and its interactions; energy) but also motivation (e.g., attitude; interest) and feelings toward chemistry (e.g., emotions; ACS, 2018; European Commission, 2015; Forsthuber et al., 2011; NRC, 2012a, 2012b, 2014, 2016; Schola Europaea, 2019). Theoretically, GBL can impact chemistry learning by affecting cognitive processes, motivation to learn, and/or emotion (NRC, 2011a; Plass et al., 2015; Plass et al., 2020). Design features of GBL match multiple theories of learning (cognitive, motivational, and emotional) to a larger extent than other media or methods (e.g., studying a web lecture). We developed the general theoretical framework of GBL (see Figure 1). Features of GBL include instructional design features (e.g., teacher support, task design, and peer interactions) and game design features (e.g., the design of the interface, number of sessions, and challenges) which are further divided into essential game design features (e.g., rules) and nonessential game design features (e.g., narratives). To solve the aforementioned challenges regarding low motivation to learn chemistry, we focus on motivation to learn a subject (e.g., chemistry) instead of motivation to play the game.

1.2 The present study

Although GBL and chemistry education align, overview research regarding the learning effects and the determining factors is limited. Despite focusing on chemistry education, this meta-analysis builds on and considers some limitations from previous meta-analyses. First, we focus on unresolved issues: whether GBL is more motivating (Sitzmann, 2011; Vogel et al., 2006; Wouters et al., 2013) than non-GBL (i.e., any type of learning activities without using games, such as learning with lectures) and which design features enhance GBL (Chen et al., 2020; Clark et al., 2016; Tsai & Tsai, 2020; Wouters & van Oostendorp, 2013). Second, to avoid deviating definitions of key concepts including cognitive gains (Vogel et al., 2006), motivation (Clark et al., 2016; Lamb et al., 2018), or simulation games (Riopel et al., 2020; Setiawan & Phillipson, 2019; Sitzmann, 2011), we define GBL narrowly by the aforementioned essential game design features such as play (i.e., exclude pure simulations) and classify learning outcomes into cognitive, motivational, and emotional outcomes. Third, as learning content varies, game genres vary and, consequently, game effectiveness may also vary (Wouters et al., 2013). Given that game genre is critical in game design and depends on to-be-learned knowledge, we examine it and follow Chen et al. (2020) and Ke's (2016) classification of game genre such as role-playing (see Table 2). Fourth, some studies only included attitude (Vogel et al., 2006), self-efficacy (Sitzmann, 2011), interest, and engagement (Wouters et al., 2013) as motivational outcomes, whereas we include other motivation theories, such as expectancy-value theory and achievement goal theory. Fifth, the standard (two-level) meta-analytic model used by previous meta-analyses did not consider dependency of effect sizes within studies (e.g., one study reported multiple comparisons, or multiple measurements for the same outcome; Cheung, 2014, 2019; López-López et al., 2018). Instead, we use a three-level meta-analytic model to estimate between- and within-study variance. Sixth, although publication bias (i.e., studies with statistically significant or positive results tend to be published more often than those with not statistically significant or negative results; Rosenthal, 1979) is one of the biggest issues in meta-analyses (Fernández-Castilla et al., 2021), some studies missed many commonly used methods when detecting and correcting publication bias (e.g., Riopel et al., 2020; Sitzmann, 2011), such as the Egger test which has been shown a better correction for publication bias than other methods (Stanley & Doucouliagos, 2014; for detailed comparison of methods, see Fernández-Castilla et al., 2021; Kromrey & Rendina-Gobioff, 2006). Progress in statistical analysis techniques enables us to use more recent and reliable meta-analysis methods, such as meta-regression to detect publication bias and investigate continuous moderators such as sample size.

This meta-analysis systematically synthesizes all experimental studies that applied GBL in K-16 chemistry education by addressing the following questions:

For cognitive outcomes, GBL changes academic knowledge, which is often measured by immediate and/or delayed tests (Mayer, 2020). As learning and instruction seek to promote the storage of learned knowledge in long-term memory that can be retrieved when needed (Bennett & Rebello, 2012; Paas & Sweller, 2014), delayed tests are advocated to determine the long-lasting impact of GBL (i.e., long-term retention) instead of fleeting knowledge improvement due to arousal (Mayer, 2014b). Furthermore, retention is a key learning outcome in chemistry education (NRC, 2012a, 2014, 2015). For motivational outcomes, that GBL are motivating is the most frequent appeal of GBL (Malone, 1981; Plass et al., 2015; Wouters et al., 2013). For emotional outcomes, decreasing boredom and increasing enjoyment is another appeal of GBL (Loderer et al., 2020). Previous meta-analyses generally found small to large effect sizes for the different cognitive outcomes (Karakoç et al., 2020; Lamb et al., 2018; Riopel et al., 2020; Setiawan & Phillipson, 2019; Sitzmann, 2011; Tsai & Tsai, 2020; Wouters et al., 2013) and retention (Riopel et al., 2020; Sitzmann, 2011; Wouters et al., 2013) in favor of GBL relative to non-GBL but disagree about the effects on motivational outcomes (Sitzmann, 2011; Vogel et al., 2006; Wouters et al., 2013), and little is known about emotional outcomes (see Table 1). Fortunately, a meta-analysis on emotions in technology-based learning concludes enjoyment and curiosity are positively related to achievement in GBL (Loderer et al., 2018). Based on this evidence, we hypothesize the following:

We identified the following instruction characteristics as moderators based on the aforementioned cognitive foundations of GBL and inconclusive results from previous meta-analyses in GBL (see Table 2). First, activity level of control group. According to CTML (Mayer, 2014a, 2014b, 2020) and CTL (Sweller et al., 2019), GBL fosters generative processing in which learners actively engage in selecting, organizing, and integrating new information, but this is also true for non-GBL. Active processing is key to learning; the deeper the processing of information, the more that will be retained and encoded into memory (Craik & Lockhart, 1972); thus, the difference between GBL and non-GBL may decline when non-GBL uses active instead of passive instruction. Second, additional instructions (i.e., instructions that are used together with GBL rather than non-GBL, such as pretraining before GBL). Organizing and integrating information is critical for learning, but these do not occur automatically (Mayer, 2014a, 2014b). Integrating GBL with non-game instructions (e.g., pretraining;Clark et al., 2016 ; Wouters et al., 2013) may facilitate articulating and integrating new knowledge with prior knowledge, leading to higher recall, transfer, and retention than standalone GBL (Merrill, 2012; Wouters et al., 2008; Wouters et al., 2013; Young et al., 2012). Third, user grouping. Playing games in groups and explaining things to each other may also facilitate knowledge articulation and, thus, the organization and integration of new information. This point is supported by the collaboration principle in multimedia learning, also known as the collective working memory effect (Kirschner et al., 2009, 2011), which states it is better to assign complex learning tasks in groups (van Merriënboer & Kester, 2014). Fourth, number of game sessions. GBL can be complex for novices. When they start playing a game, they must learn technological knowledge—game information (extraneous processing because it does not contribute to learning) and content knowledge (essential and generative processing); thus, they may easily become overwhelmed. Multiple game sessions allow the players to get familiar with the game.

Theoretically, the efficacy of GBL relative to non-GBL may improve in specific learning arrangements—such as additional instruction, group gameplay, multiple sessions, or passive instruction in non-GBL—that facilitate information processing. Empirically, previous meta-analyses only agree on the role of number of game sessions; that is, compared with non-GBL, learners benefited more when GBL involved multiple sessions, and no difference was found between single sessions and non-GBL (Clark et al., 2016; Wouters et al., 2013). However, the meta-studies are unsure regarding additional instruction (Clark et al., 2016; Sitzmann, 2011; Wouters et al., 2013), user grouping (Tsai & Tsai, 2020; Vogel et al., 2006; Wouters et al., 2013), and activity level of control group (Riopel et al., 2020; Sitzmann, 2011; Wouters et al., 2013). Therefore, we only formulated a hypothesis regarding the number of game sessions. For other variables, we investigated whether and to what extent they moderate the overall effect.

To check study quality, we included the following methodology characteristics as moderators: randomization, sample size, publication source, and assessment type (see Table 2). Ideally, large sample sizes and randomized controlled trials (RCTs) are recommended by review organizations, such as What Works Clearinghouse (WWC, 2019). Practically, effect sizes were found to be systematically higher in quasi-experiments designs (QEDs) than in RCTs, in smaller than larger studies, and in published studies than gray literature due to methodological weaknesses and small-study effects (Cheung & Slavin, 2016; Slavin, 2008; Slavin & Smith, 2009). Although closed assessment (e.g., multiple-choice questions) is easier to implement than non-closed assessment (e.g., open-ended questions), effect sizes seem larger when using non-closed assessment (Tsai & Tsai, 2018). Again, previous GBL meta-analyses provide inconclusive results on the moderating effects of these methodology characteristics (Karakoç et al., 2020; Riopel et al., 2020; Sitzmann, 2011; Tsai & Tsai, 2018; Wouters et al., 2013). Therefore, it is valuable to check publication bias. We further evaluate the extent of bias, estimate unbiased effects, and suggest improvements for future research; simply excluding unpublished studies would ignore publication bias and overestimate the overall effects.

As displayed in Table 3, value-added research pinpoints design features that promote GBL by reducing extraneous processing (e.g., redundancy), managing essential processing (e.g., modality), and/or fostering generative processing (e.g., personalization; Mayer, 2020). Previous meta-analyses have suggested that some instructional design features such as modality, personalization, feedback (Tsai & Tsai, 2020; Wouters & van Oostendorp, 2013), competition (Chen et al., 2020), and/or enhanced scaffolding (e.g., personalized scaffolding based on individual learner needs; Clark et al., 2016) enhance GBL. However, other features remain unsettled, including game design features such as narrative (integrate a storyline; Wouters & van Oostendorp, 2017) or immersion (use VR), and instructional design features such as collaboration (play in groups; Clark et al., 2016), learner control (allow learners to choose game levels), or segmenting (break the materials into parts; Mayer, 2020). Given the limited research evidence, we do not formulate a hypothesis but explore the efficacy of these features in chemistry GBL.

2.1 Literature search

The PRISMA flow diagram (Moher et al., 2009) in Figure 2 summarizes the process of the literature search and selection. The search comprised three parts. First, searching databases: Web of Science, Scopus, Eric, and PsycINFO. The search terms were combined in English: “(chemistry or chemical) AND (game or immersive learning or virtual reality or virtual environment or augmented reality or augmenting reality or mixed reality) AND (learning or cognit* or achievement or interest or attitude or engage* or motivation* or involvement or enjoy* or emotion* or affective)”, which should be in the title, abstract, or the list of keywords. In case the term “game” was not in the title, abstract, or keywords, we included immersive learning technologies commonly used in games as the search terms, such as VR. The period searched was from 2000, when GBL studies changed dramatically after that due to technological development (Parker et al., 2008) and GBL started becoming popular in chemistry, to January 2020. Second, a Google Scholar search for gray literature (Haddaway et al., 2015). Finally, a snowball search in the reference lists and citations of the aforementioned meta-analyses and systematic reviews. Overall, 1156 records came out, and 842 remained after removing duplicates.

2.2 Inclusion and exclusion criteria

Studies were evaluated based on (a) language, (b) subject, (c) participants, (d) accessibility, (e) comparison, (f) independent variable, (g) dependent variable, (h) data, and (i) others (Table 4). Study inclusion followed two stages (Figure 2). First, in the screening stage, all the titles and abstracts were screened based on a, b, f, and h, which led to 319 records being included and 523 excluded. If the study did not state whether it fulfilled these four criteria, the researchers included it and made a further decision based on its full text in the next stage. To assess the inter-rater reliability (IRR), 91 records (>10% of 842) were randomly chosen and screened independently by the first two authors, with Cohen's k = 0.83. Any disagreements were resolved by discussing and consulting with the third author. Second, the full texts of 319 records were retrieved and evaluated using all nine criteria. The first two authors randomly selected 34 full texts (>10% of 319) and assessed them independently. Despite two disagreements regarding the reason for exclusion, a perfect IRR was reached (Cohen's k = 1). In total, we included 34 articles.

2.3 Coding

To conduct a quantitative analysis and provide a qualitative description of all the included studies, we collected the following data: basic information, game genre, learning outcomes, and moderator variables (Table 2). The coding process was performed in a standardized way. First, a trial coding with five studies was run to evaluate whether all possible situations for a moderator variable were covered by a category. Then, a random sample of four studies (>10% of 34) was coded independently by the first two authors. A satisfactory IRR with Cohen's k = 1 was reached for all variables, except for number of game sessions (Cohen's k = 0.5). Any disagreements were discussed until agreement was reached. For this variable, another four articles were randomly chosen and coded by the first two authors with IRR of Cohen's k = 1. Finally, the first author coded the remaining studies. A detailed coding for moderators is available in Table S1.

2.4 Calculating effect sizes

Standardized mean difference (mean difference between experiment groups and control groups divided by the pooled standard deviation) was adopted as effect size (Borenstein et al., 2009), and g (Hedges, 1981) was calculated using Comprehensive Meta-Analysis (CMA v3; Borenstein et al., 2013). The effect size for each study was computed in a hierarchical order: the first is raw data (mean and standard deviation) and the second is data from inferential statistics (e.g., t or F value; Wouters et al., 2013).

The following complex cases were evaluated cautiously. First, when studies used a pretest–posttest control group design (k = 23), the preexisting difference between the experimental and control groups should be considered, and thus, the effect size was estimated on both pretest and posttest data (Morris, 2008). In practice, effect sizes were computed by post-pre mean difference of experimental group minus post-pre mean difference of control group divided by the pooled standard deviation of post-scores. In addition, when different sample sizes in the pretest and posttest were reported, the smaller sample size was adopted (da Silva Júnior et al., 2018).

Second, when studies used multiple experimental or control groups (k = 5), the recommended solution was to form multiple pairwise comparisons and calculate multiple effect sizes (e.g., GBL vs. concept mapping and GBL vs. conventional lecture; Okonkwo, 2012).

Third, when studies recorded multiple measurements of the same outcome (k = 8), multiple effect sizes were calculated, as suggested by Cheung (2014, 2019) and López-López et al. (2018)). For instance, knowledge comprehension and knowledge application that were assessed separately were calculated separately as the indicator for cognitive outcomes (e.g., Chen et al., 2014; Chen & Liao, 2015). Intrinsic motivation, self-determination, self-efficacy, grade motivation, and career motivation were calculated separately as the indicator for motivation in chemistry (e.g., Meesuk & Srisawasdi, 2014; Srisawasdi & Panjaburee, 2019).

Another special case was Johnson-Glenberg et al. (2014) using the AB-BA design: Two groups were given GBL intervention (A) and regular instruction (B) in different sequences. Group 1 received pretest, SMALLab GBL intervention, mid-test, regular instruction, and posttest, while group 2 received regular instruction first after pretest and SMALLab GBL intervention after mid-test. When the mid-test was conducted, groups 1 and 2 had only received GBL intervention (SMALLab) and regular instruction, respectively. Therefore, the treatment before the mid-test was considered as a single-pair comparison (group 1 as GBL group and group 2 as control group) and the mid-test instead of the “posttest” at the end of the study was taken as the posttest.

2.5 Data analysis

All statistical analyses were run via the “metafor” package (version 3.1.8; Viechtbauer, 2010) in R (version 4.1.0), except the distribution of variance across levels that was run via the “dmetar” package (Harrer et al., 2019). Because some studies reported multiple measures of the same construct (e.g., cognition was measured by knowledge comprehension and knowledge application) or multiple comparisons (e.g., two GBL groups vs. one non-GBL group), multiple effect sizes could arise per study and these effect sizes are dependent within studies. Separate meta-analyses were performed for cognition, retention, and motivation using the random-effects three-level meta-analytic model (Cheung, 2014, 2019; López-López et al., 2018). The three-level meta-analytic model includes sampling variance (level 1), within-study variance (level 2), and between-study variance (level 3). Heterogeneity was assessed using Cochran's Q test, and I² and τ² statistics. We used the Knapp and Hartung adjustment (Knapp & Hartung, 2003) to control the Type I error rate (Viechtbauer et al., 2015) and the restricted maximum likelihood method (López-López et al., 2014).

Following previous meta-analyses, the interpretation of the magnitude of the overall effect size was based on the benchmark identified by Cohen (1988): 0.2 = small, 0.5 = medium, and 0.8 = large, although his standard has some limitations such as not considering methodological features (Cheung & Slavin, 2016; Lipsey et al., 2012). Considering these limitations of Cohen's (1988) criteria, the magnitude of effect size of individual studies was also evaluated based on the benchmarks identified by Cheung and Slavin (2016): 0.30 = average for studies with small sample size (<250) and 0.16 = average for studies with large sample size (≥250).

Sensitivity analysis was conducted by checking whether the study's confidence interval overlaps with that of the pooled effect size and calculating standard deviations (z ≤ −3 or z ≥ 3.0 are outliers). Publication bias was visualized by plotting the observed standardized mean differences against their standard errors and tested by funnel plot test (using sample size as a predictor of effect sizes; Macaskill et al., 2001), Begg's rank correlation test (using variance and sample size as a predictor of effect sizes; Begg & Mazumdar, 1994), trim-and-fill method (using L₀⁺ as the number of unavailable effect sizes due to publication bias; Duval & Tweedie, 2000a, 2000b), and an adapted version of Egger's regression test (using sampling variance as a predictor of effect sizes; de Jong et al., 2021; Egger et al., 1997; Fernández-Castilla et al., 2021; Knapp et al., 2017; Sterne et al., 2011; Viechtbauer, 2017). The existence of publication bias is indicated by a statistically significant test and L₀⁺ > 3 (see Fernández-Castilla et al., 2021). The adapted version of Egger's regression test models a quadratic relationship between the standard errors and the standardized mean differences and the intercept of this model is the overall effect size free of publication bias (see Stanley & Doucouliagos, 2014).

Following de Jong et al. (2021) and Knapp et al. (2017), a three-level mixed-effects model was run for moderator analysis. Two groups of moderators were analyzed to decrease the risk of making a Type I error or Type II error caused by testing moderators individually or simultaneously (Jansen et al., 2019; van Alten et al., 2019). Due to missing values, the activity level of control group, additional instruction, and assessment type were omitted from group analysis and analyzed individually.

3.1 Descriptive findings

This meta-analysis included 34 studies published from 2006 to 2020. Their characteristics are in Table 5. The sample sizes ranged from 40 to 470. Thirty studies used media comparisons, while only three made value-added comparisons. For learning outcomes, no study reported emotions, nine reported motivation, 33 reported cognition, and six reported both cognition and motivation, but only three reported both an immediate test and a delayed test (retention). For educational level, all studies were implemented in secondary schools (k = 21) and universities (k = 13). Regarding country, one third was conducted in the United States (k = 11) and one in eight in China (k = 4). For chemistry content, the most common topics were nomenclature (k = 8), periodic table (k = 4), and organic chemistry (k = 4). Regarding assessment methods, tests and questionnaires were the most frequent measures for cognition and motivation; only five studies adopted mixed-method research (e.g., tests combined with interviews), among which one retrieved log data. Regarding game genre, the most used genres were puzzle (k = 12), simulation (k = 7), and role-playing games (k = 6). A detailed example of GBL activities for each game genre is available in Table S2.

3.2 Research question 1: Media comparison

3.2.2 Distribution of effect sizes

Effect sizes of cognition and motivation for the individual studies and their distribution are presented in forest plots (Figures 3 and 4). No results were found for emotion. As displayed in Table 6, regarding cognitive outcomes, the effect sizes (k = 30, #ES = 57) vary substantially across the studies, from −0.62 to 1.84. Among the 53 positive outcomes, 40 are statistically significant. Among the five large-scale studies (sample size ≥250), three reported an equal or above average effect size (≥0.16) and among the 52 small-scale studies (sample size <250), 46 reported above average effect size (≥0.3), according to Cheung and Slavin (2016). The mean effect size is statistically significant and medium (g = 0.70; 95% CI [0.51; 0.89]), according to Cohen (1988). Regarding retention, a similar effect (g = 0.59; 95% CI [0.35; 0.83]; k = 20, #ES = 31) is found. Regarding motivation, the effect sizes (k = 7, #ES = 21) vary from −0.09 to 1.18. Among the 19 positive outcomes, eight are statistically significant. Only two reported a negative but not statistically significant effect; the mean effect is statistically significant but small (g = 0.35; 95% CI [0.19; 0.50]). The model fit of this three-level model was statistically significantly better than the two-level model than the two-level model that does not consider within-study variance (cognition: χ² = 17.20, p < .001; retention: χ² = 5.88, p = .01; and motivation: χ² = 6.88, p = .009).

TABLE 6. Results of random-effects meta-analysis in media comparisons

Variable	k (N)	#ES	g	SE	95% CI	Q	τ2level2	τ2level3	I2level2	I2level3
Cognition	30 (4155)	57	0.70	0.10	[0.51; 0.89]	415.2*	0.10	0.16	33%	53%
Retention	20 (2860)	31	0.59	0.12	[0.35; 0.83]	240.4*	0.06	0.22	18%	70%
Motivation	7 (974)	21	0.35	0.08	[0.19; 0.50]	49.14*	0.05	0.01	49%	12%

• Notes: CI, confidence interval; #ES, number of effect sizes; g, mean effect size; k, number of studies; N, total sample size; Q, heterogeneity value; SE, standard error; τ²_level2, within-study variance; τ²_level3, between-study variance; I²_level2, within-study heterogeneity index (%); I²_level3, between-study heterogeneity index (%).
• * p < .05.

3.3 Research question 2: Moderator analysis

As displayed in Table 7, at least one moderator in methodology characteristics exhibits a statistically significant relationship with the effect size (p < .0001). Specifically, sample size is negatively related to effect size (estimated coefficient = −0.003, p < .0001), and publication source is positively related to effect size (estimated coefficient = 0.62, p = .001), while accounting for all the other variables in the model, that is, small-scale studies or published studies are associated with larger effects. Aside from these two variables, no other moderator exhibits a statistically significant relationship with the effect size.

3.4 Research question 3: Value-added comparison

As displayed in Table 8, only three studies reported six effect sizes for cognition, ranging from −0.11 to 0.46, but none reached statistical significance. The only statistically significant but very small effect is in the study that compares the effect of worked examples on motivation. A meta-analysis was impossible as the comparisons differed strongly, ranging from manipulations of aesthetic, choice, competition, worked example, guiding strategy, and type of AR.

4.1 Media comparison

Our first goal was to examine whether chemistry GBL has a larger effect on cognitive, motivational, and emotional outcomes than non-GBL (RQ1). Most studies focused on cognitive outcomes, providing promising evidence that GBL enhances chemistry learning, but did not include motivational outcomes, providing moderate evidence that GBL motivates interest in chemistry. No evidence is available on whether GBL increases positive emotions or decreases negative emotions as no study reported emotional outcomes. Compared with previous GBL meta-analyses, this study is the first meta-analysis that uses a three-level random-effects model to consider the dependency of effect sizes within studies and the first that emphasizes emotion in GBL.

First, this study confirms chemistry GBL is more effective for cognition (Hypothesis 1) and retention (Hypothesis 2) than non-GBL. The mean effects for cognition (g = 0.70) and retention (g = 0.59) reveal a statistically significant medium (g > .5; Cohen, 1988). In other words, the score of the average person in chemistry GBL would be 0.6 SD above non-GBL, exceeding 73% of students in non-GBL (Coe, 2002). The effect for cognition is larger than most previous GBL meta-analyses across all subjects in general and math and science in particular, but equal or smaller than those in English (Table 1). This effect is also comparable with previous meta-analyses particular to chemistry with other educational interventions (cooperative learning: g = 0.59, Apugliese & Lewis, 2017; g = 0.68, Warfa, 2016; cooperative learning, collaborative learning, problem-based learning, process oriented guided inquiry learning, peer-lead team learning, and flipped instruction: d = 0.62, Rahman & Lewis, 2020). Most importantly, the effect for retention is larger than all previous meta-analyses. This implies that 0.59 could be a benchmark for a meaningful effect in chemistry education.

Three reasons could explain the differences in the magnitude of the overall effects between current and prior meta-analyses. One reason is that chemistry has a special relationship to GBL: GBL better align with the key characteristics of chemistry education (see Introduction) than other subjects. If that is the case, policymakers and practitioners should implement GBL in chemistry education. Second, technology development: more sophisticated technologies improve learning. Studies included in this meta-analysis, published from 2006 to 2020, are more recent than those from previous meta-analyses, ranging from 1990 to 2012 (Clark et al., 2016; Wouters et al., 2013). During the past decade, new technologies have emerged (Chen, Wang, et al., 2018). Our included studies applied many sophisticated technologies. For instance, based on voice recognition, eye movement, and brain wave analysis, Natural User Interface is used in gaming consoles (Jagodziński & Wolski, 2015); real-time data capture system is used in blended reality environment (Hodges et al., 2018); different types of automated tutoring based on student performances are combined with interactive dialogs with avatars (Halpern et al., 2012); VR simulates experiential learning (Su & Cheng, 2019); and MR benefits embodied learning (Johnson-Glenberg et al., 2014). These sophisticated technologies may better support chemistry GBL. Furthermore, students now have better access to technologies, leading to less difficulty playing chemistry games. Third, with the development of instructional design, current chemistry GBL may be better embedded in learning theories than older ones. More attention is paid on integrating game design and instructional design when designing chemistry GBL (e.g., Mayer, 2014b; NRC, 2011a; Plass et al., 2015) as most studies are from a later period. Nevertheless, chemistry GBL can enhance cognition, and the effect lasts over time.

This study also suggests chemistry GBL is more motivating than non-GBL (Hypothesis 3). Different from Wouters et al. (2013; d = 0.26, p > .05), a small but statistically significant effect (g = 0.35; g > 0.2; Cohen, 1988) for motivation was found. In other words, the motivation score of the average person in chemistry GBL would be 0.4 SD above non-GBL, exceeding 62% of students in non-GBL (Coe, 2002).

This finding seems to refute the critique that GBL may attract students, but higher motivation does not necessarily mean higher learning. Even though students report liking or having interest in the medium (the game), they tend to perceive that it provides an easier path to learning and invest less mental effort and time (Salomon, 1984), resulting in less learning compared with learning without the medium (Clark & Feldon, 2005, 2014). In our case, two included studies confirm this critique: students prefer GBL to study guides (Wood & Donnelly-Hermosillo, 2019) or traditional lectures (Stringfield & Kramer, 2014), but no difference in achievement was found. However, two other studies support our finding that GBL promotes both achievement and motivation to learn chemistry (Cha et al., 2017; Srisawasdi & Panjaburee, 2019). Nevertheless, given that only seven studies reported motivation, this result should be interpreted with caution.

The cognitive and motivational benefits of chemistry GBL cannot prove a causal relationship between cognition and motivation. In the studies, only five reported both outcomes and their research methods, one-time pre-posttest design or posttest-only design, may not provide the required evidence. Instead, the cross-lagged panel model aims to detect causal or reciprocal relationships between variables, analyzing longitudinal data collected by testing or recording subjects at multiple points over time (Hamaker et al., 2015; Mulder & Hamaker, 2021; Selig & Little, 2012). In our included studies, no such method was used. Thus, whether cognition causes higher motivation in chemistry GBL and whether motivation causes higher cognition remains open questions.

4.2 Moderator analysis

Our second goal was to examine the possible moderating effects of instruction and methodology characteristics, that is the conditions under which GBL is more effective relative to non-GBL (RQ2). We found some evidence that methodology characteristics moderate the effects, particularly sample size and publication source. Compared with previous GBL meta-analyses, this study uses more advanced methods to detect and correct publication bias and includes a continuous moderator (i.e., sample size).

Larger effects may be associated more with published studies than gray literature and with smaller studies than larger ones. The small-study effects, particularly publication bias, tent to exist. Researchers in chemistry education and GBL should attend to this issue, given that similar findings were also reported by previous meta-analyses particular to chemistry with other educational interventions (e.g., Rahman & Lewis, 2020; Warfa, 2016) and by meta-analyses in GBL (e.g., Lamb et al., 2018; Riopel et al., 2020; Sitzmann, 2011). However, more standardized methods with high statistical power are needed to assess and control how they impact main effects (e.g., the trim-and-fill method imputes adjusted effect size) and other aspects in multilevel meta-analyses (P. Cuijpers, personal communication, April 20, 2020). For instance, should we add sample size or publication source as covariate of the main effect? How and to what extent do small-study effects influence the moderator analysis?

Other moderators did not reveal statistically significant effects. Effect sizes of cognition were equal between non-GBL with active vs. passive instructions, GBL with vs. without additional instructions, GBL with single vs. multiple sessions (Hypothesis 5), GBL individually vs. in groups, RCTs versus QEDs, or with closed question vs. non-closed questions. Given the small number of studies under moderator categories, these results should never be interpreted as evidence that the effects are the same across subgroups or that there is no relation between the effects and included moderators (Borenstein et al., 2009). Instead, further studies are needed for more reliable evidence. Take randomization, for example, it is premature to conclude that larger effects are associated with RCTs than QEDs based on six RCTs versus 24 QEDs. Moreover, it is difficult to explain why we did not find statistically significant results for those moderators due to the limitations of all meta-analyses.

Other variables may help explain the potential sources of between-study variance. Unfortunately, the number of studies in total or under each subgroup was too low to conduct a moderator analysis. Instead, we performed an explanatory analysis based on findings from specific studies. One potential moderator is game genre. A specific game genre may suit specific chemistry content (Wouters et al., 2013). For instance, puzzle games may help build factual knowledge (e.g., nomenclature, g = 1.8; Chimeno et al., 2006) through strengthening and weakening associations (reinforcement theory; Skinner, 1938); simulation games may help build conceptual knowledge (e.g., redox reaction, g = 0.61; Hodges et al., 2018) through constructing a schema of the cause-and-effect system (schema theory; Paas & Sweller, 2012); simulation games with MR or VR may help build procedural knowledge (e.g., titration, g = 0.97; Johnson-Glenberg et al., 2014) through deliberate practice with feedback (automaticity theory; Fitts & Posner, 1967; Mayer, 2014b). However, which genres suit which types of chemistry knowledge for which types of learners and under which contexts remains to be explored.

Another potential moderator is individual difference, such as gender (e.g., Steegh et al., 2021), prior knowledge (e.g., Lou & Jaeggi, 2019), and prior game experience. Among our included studies, compared with non-GBL, (1) girls outperformed boys but were not more motivated in chemistry GBL (Okonkwo, 2012), whereas others found no gender difference (Hodges et al., 2018; Merchant et al., 2013; Weng et al., 2015); (2) students with lower prior knowledge experienced greater learning gains from GBL than those with higher prior knowledge (Merchant et al., 2013; Wood & Donnelly-Hermosillo, 2019), but others found no difference (Sousa Lima et al., 2019); and (3) students with game experience achieved slightly higher learning gains than those without game experience (Merchant et al., 2013).

4.3 Value-added comparison

Our third goal was to identify the more effective game design and instructional design features for chemistry GBL (RQ3). However, studies that used value-added comparisons of GBL with or without specific features (k = 3) are too few to perform a meta-analysis. This lack of studies confirms that the study of effective design features of GBL (value-added research) is often underestimated compared with media comparison research (Boyle et al., 2016; Clark et al., 2016; Young et al., 2012). In line with previous meta-analyses, more evidence from value-added research is required for researchers and practitioners.

First, value-added research may provide design guidelines for chemistry GBL, especially for practitioners such as game developers who create games for learning and teachers who implement GBL (Mayer, 2014b). GBL can be complex and require well-designed guidelines (e.g., Eastwood & Sadler, 2013). There are little evidence-informed guidelines for developers to integrate instructional design with game design features. Most game developers are familiar with game design but not instructional design. However, most teachers can only change the GBL environment by instructional design, not the game environment per se. Second, game researchers must first conduct value-added research to refine GBL environments before comparing GBL with non-GBL. Without optimizing GBL through value-added comparisons, it is unpromising to compare learning with poorly designed games versus other media (Plass et al., 2020).

According to one side of the Clark-Kozma “media-effects” debate, media comparison studies come with two challenges. Conceptually, research may confound media (games) with methods; it is not the medium but the method that causes learning (Clark, 1983, 1991, 1994a, 1994b, 2007; Clark et al., 2008; Kirschner & Hendrick, 2020; Mayer, 2014b). Methodologically, GBL and non-GBL groups may differ in dimensions (e.g., instructions, learning materials) other than the game, making it unclear what makes a difference in learning (Clark, 2007; Clark et al., 2008; Kirschner & Hendrick, 2020; Mayer, 2014b; NRC, 2011a). Therefore, it is difficult to attribute learning effects to games, instructional methods, or other factors (e.g., Daubenfeld & Zenker, 2015).

One solution is to focus on value-added research within one game. The other side of the Clark-Kozma “media-effects” debate argues that it is unnecessary to separate instructional methods from games, as together they cause learning (Kozma, 1991, 1994a, 1994b; Parker et al., 2008). Instead of separating them, a good GBL design integrates instructional design with game design. Cognitive benefits are not the sole potential of chemistry GBL as games complement learning experiences with other aforementioned unique potentials (see Introduction). To employ these potentials, the focus should be less on media comparisons regarding learning effects, and more on improving GBL via value-added comparisons.

However, this does not mean media comparisons are meaningless and should be abandoned completely; they are still valuable, especially when testing GBL superiority claims (GBL is more effective than learning other media; Mayer, 2014b), justifying the reward, the effort, and cost of developing games for learning, and verifying whether certain instruction methods work specifically for GBL but not for non-GBL. Furthermore, with media comparison research, another solution is to equate GBL and non-GBL in all variables except for the game (Mayer, 2014b). Before that, however, we need high-quality GBL. Again, value-added research comes into play. Since value-added research and media comparison research serve different functions, researchers must first conduct value-added comparisons to create a well-designed GBL and then, if necessary, conduct media comparisons using a rigorous experimental design.

4.4 Limitations

The following concerns may affect the study findings. First, some studies fail to report background information. For example, because six studies reported limited or no information about additional instructions and activity levels of control groups (e.g., Fatokun et al., 2016; Rastegarpour & Marashi, 2012), their moderating effect is unknown. Although the considered moderators captured part of high heterogeneity, there is clearly unexplained variance. Missing information prevents us from including other potential moderators, such as game experience, educational research experience, or duration of intervention (Wouters et al., 2013; Wouters & van Oostendorp, 2013). Missing information affects the selection, coding, and/or analysis of moderators. Furthermore, missing information might affect our assessment regarding study quality. For instance, GBL adopts debriefing, whereas this information is missing in non-GBL, or two groups may use different ways to present the learning content. Thus, research may be contaminated (Kirschner & Hendrick, 2020) as there are more differences between the comparison groups other than just the game (Clark, 1983). Again, we cannot include or control this influence because of the missing information.

Unfortunately, we had to exclude many studies because essential information was missing. Out of 842 screened articles, only 34 met our criteria, indicating that many chemistry GBL have been developed but are not well-reported and/or well-tested. The increasing popularity of games stimulated a flood of publication (Hwang & Wu, 2012; Tobias et al., 2011), but most of the excluded studies only describe the game without testing its effectiveness on learning outcomes (Tobias & Fletcher, 2012); report students’ subjective opinions, satisfaction, or conceptions of the game (i.e., the usability test) without an objective assessment; or measure learning outcomes without a control group. Although post hoc power analysis is not recommended, it indicates we have sufficient power for cognition, retention, and motivation benefits of chemistry GBL.

Moreover, our broad definition of cognitive and motivational outcomes may bias the main effects. Studies vary regarding outcome measures (Cooper, 2015), and the limited number of studies made it impossible to categorize them further into different constructs (e.g., interest, self-efficacy). As all studies are different, the focus is less on sameness and more on difference: what makes effect sizes varied (Hattie, 2013). In this sense, for motivation, the meta-analysis indicates the effects of the interventions on motivation did not vary across type of motivation (Lazowski & Hulleman, 2016). For cognition, previous meta-analyses on GBL also imply the effects did not vary across type of cognitive outcomes: knowledge vs. skills (Wouters et al., 2013), declarative vs. procedural knowledge (Riopel et al., 2020; Sitzmann, 2011), or intrapersonal vs. cognitive learning outcomes (Clark et al., 2016; see Table 1).

4.5 Implications

We make the following recommendations regarding the practices and theories of chemistry GBL. For practitioners, the positive effect of chemistry GBL on cognition, retention, and motivation suggests implementing GBL in chemistry education. The overall effect size provides the benchmark of chemistry GBL interventions for further research, and the distribution of effect sizes may help researchers anticipate the effects before their study.

For researchers, we agree with previous meta-analyses that it is time to move beyond “whether or not chemistry GBL works” (media comparison) to “what works for chemistry GBL and why it enhances learning and others do not” (value-added comparison; Chen et al., 2020; Young et al., 2012) and conduct more research to provide design guidelines for implementing chemistry GBL. This meta-analysis suggests that GBL may address the unique characteristics of chemistry. To confirm this, more GBL meta-analyses on subjects other than chemistry are needed.

Regarding learning outcomes, more considerations are needed: (1) for cognitive outcomes, more delayed tests measuring retention (Mayer, 2014b; Wood & Donnelly-Hermosillo, 2019) to avoid the novelty effect (Clark, 1983); (2) more research in motivation to learn chemistry, which could be a common but questionable appeal of chemistry GBL (Clark & Feldon, 2005, 2014); (3) more research into emotions (e.g., Raker et al., 2019) since the most desirable instruction is that learners learn most from what they enjoy most (Clark, 1982); (4) further research regarding which game genre is best for which type of learning outcome; and (5) more research on the relationships between cognition, motivation, and emotion (e.g., Gibbons & Raker, 2019) in GBL.

Regarding methodology, more high-quality intervention research in chemistry GBL is required to identify what works, for whom, and under which conditions. Considering the small-study effects, we suggest researchers to use power analysis (e.g., G*Power; Faul et al., 2007) to estimate the minimum number of participants needed (Ellis, 2010) when planning primary studies as statistical power and effect size depend on sample size (Simpson, 2017). Considering the contextual factors, mixed methods are promising to evaluate the effectiveness of chemistry GBL; tests or questionnaires should be combined with observations, interviews, and/or log data (e.g., Hodges et al., 2018; Wood & Donnelly-Hermosillo, 2019). Regarding assessments, all assessments of the included studies were taken postintervention using separate tests, such as self-reports on motivation after gameplay when motivation might decrease (Wouters et al., 2013). We advocate more embedded tests (e.g., stealth assessment for adaptivity; Shute et al., 2017) or real-time overt assessments (e.g., eye tracking, physiological monitoring, heart rate, blood pressure; Mayer, 2020; Wouters et al., 2013) focusing on learning processes.

On the theoretical aspect, more evidence is needed regarding how people learn chemistry with games (learning mechanics) and how to support chemistry GBL (instructional support). First, we lack studies on how chemistry games affect learning processes and outcomes; how they affect motivation or emotions; what the roles of motivation and emotions are; and how motivation, cognition, and emotions interact with each other. Second, we lack studies on how people learn better with chemistry games. As GBL is part of multimedia learning, game researchers can refer to multimedia principles from CTML (Mayer, 2014a). In certain subjects, some principles enhance GBL (e.g., modality), whereas some do not (e.g., redundancy; Mayer, 2020). Future studies should explore the learning effects of these multimedia principles in chemistry GBL.