Immersive virtual reality for supporting complex scientific knowledge: Augmenting our understanding with physiological monitoring

Learning with immersive virtual reality

Research in the educational applications of IVR for K-12 and higher education can be summarized by several key reviews and meta-reviews surveying the field (Jensen & Konradsen, 2018; Merchant et al., 2014; Radianti, Majchrzak, Fromm, & Wohlgenannt, 2020). A core set of prior research on immersive VR considered how to design applications in order to support scientific conceptual understanding (eg, Dede, Salzman, Loftin, & Ash, 2000; Moher, Johnson, Ohlsson, & Gillingham, 1999; Roussou, Johnson, Moher, & Virtual, 1999). In the ScienceSpace project, three virtual worlds (NewtonWorld, MaxwellWorld, PaulingWorld) were created with a learner-centered design strategy, undergoing iterative cycles of design and evaluation on various aspects of the learning experiences, processes and outcomes (Dede, Salzman, & Bowen Loftin, 1996; Dede et al., 2000; Salzman, Dede, Loftin, & Chen, 1999). For example, in PaulingWorld, learners used head-mounted displays (HMDs) to explore the molecular structure and chemical bonding by manipulating different molecular representations of simple and complex molecules like water and hemoglobin. One outcome of this program of research is a model for understanding how IVR aids complex conceptual learning, which includes student characteristics that moderate learning such as gender, domain experience, spatial ability, computer experience, motion sickness history and immersive tendencies (Salzman et al., 1999). The model also considers features afforded by IVR, including 3D representations, multiple frames of references and multisensory cues. On the third affordance, Salzman and colleagues posit that users can interpret visual, auditory and haptic cues via high-end IVR interfaces, to gather information while using their proprioceptive system to navigate and control objects in the synthetic environment, citing Nugent (1982) and Psotka (1995)’s work for its potential to deepen learning and recall. However, limitations of the technology at the time and discomfort caused by head-mounted displays hindered many of the studies on usability and learning, which made systematic evaluations of learning difficult. Further study is needed to “tease out” the affordances of IVR and factors involving individual characteristics that impact conceptual understanding.

A common issue reported by researchers is that cognitive overload can hinder learning in IVR. Moreno and Mayer (2004) examined two levels of immersion, desktop computer (low immersion) or head-mounted display (high immersion) in a simulation game, Design-A-Plant, for engaging college students about the structure and function of plants. They found that immersion as a variable in this context did not appear to enhance conceptual learning and identified cognitive load to have played a limiting role. To address cognitive overload in IVR, researchers are turning to physiological measures as an alternative to existing methods (Skulmowski & Rey, 2017). A recent study examines the cognitive load directly through EEG measures (Makransky, Terkildsen, & Mayer, 2017). Fifty-two university students were offered either a desktop or IVR simulation with HMDs. Participants that engaged with the desktop version had significantly higher knowledge gain than those in the IVR group and accompanied by higher workload which supports prior concerns regarding cognitive overload in IVR. However, the use of IVR was also found to foster positive emotions and improved presence (Makransky & Lilleholt, 2018), which underscores the promises and challenges for learning offered by IVR.

Embodied cognition & sensorimotor engagement in learning

There are various forms of IVR in education ranging from 360-videos to full-body experiences, as well as different interaction paradigms (eg, view only, single button interaction on smartphone-powered headsets, to full-body tracking with bi-modal controllers). Full-body interfaces necessarily mean that users interact with technology in a more embodied, natural way—for example, in contrast to personal desktop computers or even handheld devices (Eisenberg & Pares, 2014). There is a growing body of research that supports the idea that learning an abstract subject matter is informed by bodily activity, which is associated with the study of embodied cognition (Abrahamson & Lindgren, 2014; Johnson-Glenberg, Birchfield, Megowan-Romanowicz, & Snow, 2015). Embodiment is understood as “the enactment of knowledge and concepts through the activity of our bodies” (Lindgren & Johnson-Glenberg, 2013, p. 445). Embodiment offers a lens from which activity in the full-body IVR world may be understood since the kinds of interactions are similar to the way we interact with the everyday world (Rogers, Scaife, Gabrielli, Smith, & Harris, 2002).

To advance the study of embodiment for learning, Johnson-Glenberg and colleagues (2014) developed a taxonomy, the Embodied Education Taxonomy, that posits four dimensions matched to three constructs: level of sensorimotor engagement, congruency of gestures (to learning concepts) and immersion experienced by the learner. For example, a full-body interactive simulation of planetary astronomy (Lindgren, Tscholl, Wang, & Johnson, 2016) would attain a 4^th degree in the taxonomy, which represents a high level of embodiment with high sensorimotor engagement, high congruency of gestures and high immersion. A seated desktop experience of the same simulation would attain a 2^nd degree in the taxonomy, as users are stationary and use a small screen. A 3^rd degree would entail an experience with a large or immersive display that could engage the whole-body but is generally in one place.

Our study examines two embodied conditions in IVR to complement lecture-based instruction. Focusing on the sensorimotor engagement dimension of the Embodied Education Taxonomy which relates to the magnitude of the motor signal (Johnson-Glenberg, 2018), we put participants in seated or standing and freely moving conditions within a trackable space. The seated experience maps to a 3^rd degree, whereas the standing experience maps to a 4^th degree in the taxonomy, which presumes full-body movements activate more sensory-motor neurons (ibid.).

A few studies have explored seated and standing body positions on cognition. These studies offer results on sedentary (ie, sitting) versus standing or moving for task execution. In a study testing the effect of treadmill walking speed on typing performance when these tasks were performed simultaneously, research found that among 24 research participants those seated performed better than those in motion (Funk et al., 2012). In another study, users of sit-stand desks were compared to those using sitting workstations in an office environment. Those in the sit-stand configuration showed higher productivity, however, the authors note that the study timeframe was too short to hypothesize longer term effects (Nerhood & Thompson, 2016). The results in these studies are inconclusive but do point to a connection between gross body movements and cognition. We examine the effect of sensorimotor engagement on learning in real-time, using physiological sensors and eye-tracking to acquire a continuous understanding of physiological states during the IVR experience. This complements a conventional dataset, which includes test scores, interaction logs and semi-structured interviews.

Research questions

Participants and study design

In the Winter of 2019, when the study was conducted, 245 students were registered in the course, with 234 completed the course. The study followed a between-subjects design: control group, seated IVR, standing IVR. Volunteers were recruited by a research team member, who made announcements during the lecture. Thirty-four students were randomly assigned to one of the IVR conditions. Their mean age was 21.2 years and 15 did not have prior IVR experience. We present demographic information from each group in Table 1. Compensation in the form of a gift card was provided as a token of appreciation.

Materials

IVR simulation design

MolgenVR is a virtual reality simulation environment developed for this study in Unity 3D to support students' understanding of the lac operon gene regulation system. The components of the lac operon inside a virtual E. coli bacterium served as their main view (Figure 1). Once participants have been shown how to navigate within the IVR environment (eg, teleportation), they completed several tasks, they:

• completed the lac operon by “dragging-and-dropping” genes in the correct sequence (Figure 1A)
• made a copy of the lac operon genes (ie, mRNA; Figure 1B)
• translated genetic material into proteins (Figure 1C)
• identified regulatory molecules (eg, lac repressor) and their function (Figure 1D)

Participants were asked to make predictions about individual processes related to the lac operon and the overall effect of the environmental conditions on the efficacy of the operon.

Apparatus

A retrofitted HTC Vive head-mounted display with integrated eye tracking (Tobii Pro VR, Tobii Group) was used. It was connected to a PC gaming laptop next to the HTC Vive setup in a seminar room located at Carleton University (Figure 2). A 5-point calibration is used, with eye movements recorded at 120 Hz. Physiological data were collected by CAPTIV-L7000 on a second PC laptop which synchronously records multiple sensors via a wireless receiver (T-Rec). We used the following T-Sens sensors:

• GSR: Two electrodes were positioned at the tip of the index and third fingers of the participant’s non-dominant hand. GSR data were recorded in μS (Siemens) at a frequency of 32 Hz.
• Temperature: Measurement of skin temperature. A wire probe was adhered to the palm of the participant’s non-dominant hand (below the thumb) and recorded at a frequency of 32 Hz.

Procedures

Course evaluation procedure

125 students from the course completed the LOCI (Stefanski et al., 2016), which provided a baseline understanding of the students’ incoming knowledge about the lac operon (Figure 3). Over two class periods, the course instructor gave two 80-minute lectures on the lac operon. In the two weeks that followed, the volunteers were invited to an IVR session (detailed below). Afterward, all students in the course were given a homework assignment to engage with a 2D simulation about the lac operon. On the last day of class, the LOCI was given again, of which 149 students completed. During the last two weeks of term, 12 students (two control, four seated IVR, six standing IVR) were invited for a semi-structured interview about their learning experience.

Learning outcomes

Effect of sensorimotor engagement on learning outcomes

The pre and posttests administered immediately before and after the IVR experience allowed us to understand the effect of sensorimotor engagement on learning outcomes (RQ2). This analysis addressed differences in students’ understanding of genetic regulation in standing versus seated conditions. An ANCOVA examined posttest results between the seated and standing conditions, controlling for the pretest score. A preliminary analysis evaluating the homogeneity-of-slopes assumption indicated that the relationship between the covariate (pretest score) and the dependent variable (posttest score) differed significantly as a function of the independent variable (condition), F(1, 30) = 5.77, MSE = 1.79, p = 0.02, partial η² = 0.16.

To further understand how learners’ domain experience influenced learning outcomes, we examined the effect of sitting and standing in IVR on students’ posttest performance, grouping students with different levels of prior knowledge separately. Based on the 9-item pretest scores students were classified as low-to-intermediate prior knowledge (0-6 correct, n = 15) or high prior knowledge (7–9 correct, n = 17).

An independent-samples t-test was conducted to evaluate the posttest scores for students with low-intermediate prior knowledge when seated compared to a standing IVR experience (Table 3). The test was significant, t(13) = 2.39, p < 0.05. Seated students on average scored higher than students who were standing. The 95% confidence interval for the difference in the means was quite wide, ranging from 5.55 to 44.45. The eta square index indicated that 31% of the variance of the test scores was accounted for by body position during the IVR experience. For high prior knowledge students, there was no significant difference between posttest scores for the seated group compared to those who were in the standing group, t(13) = −0.61, p = 0.55, indicating that for students with strong domain prior knowledge being seated or standing did not influence learning outcomes. Figure 4 is a plot of pretest and posttest scores grouped by low-intermediate prior knowledge and high prior knowledge. Slopes from seated conditions were associated with the higher posttest outcomes despite lower pretest scores.

Table 3. Means and standard deviation for IVR conditions and prior domain knowledge groups

Measure	Total		Low-intermediate prior knowledge		High prior knowledge
	Seated IVR	Standing IVR	Seated IVR	Standing IVR	Seated IVR	Standing IVR
	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)
Posttest score	84.31 (18.45)	81.70 (21.50)	81.11 (20.32)	55.56 (20.32)	88.89 (15.71)	92.59 (10.94)

• * Significantly higher scores compared to the standing IVR group at the p < 0.05.

Learning experience

To understand the effects of learner characteristics and sensorimotor engagement on students’ learning experience (RQ2), we analyzed the IVR sessions to explore possible cues that elicited psychophysiological responses (Table 4). Below we describe two representative cases from seated and standing IVR groups that best illustrate our findings. By comparing the cases, we hoped to detect and characterize any differences between the seated and standing IVR experience in terms of the environment, physical effort and mental effort.

Case 1: Seated IVR experience

Kara (a pseudonym) is a 2^nd year Biology major student who came into the experience with low-intermediate prior knowledge of the lac operon, according to the pretest (44%). She had high spatial ability (scoring 17 out of 20, with her peers scoring an average of 11 (M = 11.35, SD = 4.19) in comparison. She had no prior experience with IVR technologies prior to the session. Kara was seated during her IVR session and experienced increasing GSR with concurrently decreasing body temperature when the environment was first loaded onto her HMD, indicating “stress” or higher mental workload (Figure 5). The next set occurred when she was introduced to the various elements in the simulation, followed by Kara’s unsuccessful attempts at “throwing” the genetic material towards a target placeholder along the DNA strand (ie, by swinging her arm in a back to front motion, while releasing the trigger button at 5:28). If the element did not snap into place, it indicated that it was not meant to be located there or that she did not throw hard enough. Mid-session (at 17:42), GSR and temperature responses occurred when Kara was reading protein names (as they were being translated from the genetic material) and at times when objects were obstructing her view. She moved them away after a brief moment of “shrinking back” into her chair (21:08). Towards the last third of the session, areas of “stress” were consistent with Kara looking around the environment, trying to make sense of what she is seeing, explaining each step of the process and voicing the function of each component (eg, “and those [CAP protein] won’t attach there without cAMP”). Kara scored 100% on the posttest and reported no symptoms of simulation sickness. She rated her enjoyment of her experience 6 out of 7. During the interview, Kara noted that it was helpful to see the system in 3D and that she could move to different (teleportation) spots to view it from different angles. She also indicated that she preferred sitting down:

I wouldn’t want to do that standing up but also that it was like spinning chairs [referring to the swivel chair] that I could like turn around and look at everything around me…. Because I feel like I would have fallen over not being able to see [what] that’s like or run into something when I’m trying to walk around, but I don’t know I just think it forced me to not have to worry about my actual surroundings and I could worry about the VR [simulation] instead.

Case 2: Standing IVR experience

Emily (a pseudonym) is a 2^nd year Biology major student who has a high degree of prior domain knowledge (89%) according to her pretest. She has a high spatial ability (16 out of 20) and no prior experience with IVR. Emily experienced MolGenVR from a standing position. Similar to Kara, increasing GSR and decreasing temperature (ie, “stress”) was observed during the introduction of the environment (Figure 6), as well as when a gene element did not fit along the DNA strand (eg, having made an incorrect choice in putting together the structure of the lac operon at 03:36). Unlike Kara, Emily did not encounter issues with thrown genetic elements not reaching the intended target. However, she was unsure of how to interact with the genetic elements, which produced a mild physiological response. This also occurred when she was searching for a specific element but had trouble locating it. Emily did not appear to be bothered by objects obstructing her view. In the last third of her session, physiological responses occurred as Emily talked through how each genetic component functioned as she duplicated each gene and saw the resulting outcome (similar to Kara’s experience). After the session, she scored 100% in the posttest and rated the enjoyment as the highest possible score. She reported slight eye strain, slight sweating and slightly blurred vision. During the interview, Emily noted that she memorized information about the lac operon before her IVR experience and that she did not “get” the complex system until being in IVR. She cited the visual modality as a contributing factor, as well as being able to interact with the system. When asked about her experience standing in IVR she said:

I liked it I think because if I were to sit down I wouldn't be as into it as much because I felt liked it was nice to be able to turn around really easily and find the missing pieces and I felt like I would walk towards it to put it in [to the DNA]…So, I feel like it just immerses you a little bit more.

As evidenced in this quotation and in others from participants, IVR was also motivating for students to use as a supplement to the other modes of instruction. Motivation can be an important component to learning and while we do not focus heavily on this in the study, the interview responses offered some indication of interest in IVR to learn the course material.

RQ1. Effect of IVR on student understanding of complex concepts in a university undergraduate science course

First, students who participated in IVR sessions demonstrated significantly better learning outcomes statistically compared to students in the course as originally designed (control). Although the findings seemingly stand in contrast to prior studies that revealed neutral or negative results when compared with a desktop-based, alternative form of instruction (Makransky et al., 2017; Moreno & Mayer, 2004), it can point to a number of mitigating factors, including the learning content and pedagogical underpinnings that inform the IVR designs (Fowler, 2014). While previous studies indicate that users engaged with low to moderate level cognitive skills and content complexity, the current study is concerned with acquiring complex, dynamic understanding of interacting elements that go beyond simple declarative knowledge. To-date few examples of educational research examining the use of IVR to teach higher level cognitive skills exist (Jensen & Konradsen, 2018; Radianti et al., 2020). By conducting the study within an authentic learning context (ie, as part of a University course), we acknowledge that such learning is a complex process of reflective exploration. Other factors, such as positive emotions and cognitive value from their IVR experience, may have led to the improved outcomes associated with long-term motivational effects (Makransky & Lilleholt, 2018; van Merriënboer & Sweller, 2005, Moher et al., 1999; Pekrun, 2016; Ryan & Deci, 2000), however, further study is needed.

We note that all students in the course participated in the 2D simulations and that IVR participants had the additional experience. While the positive effects could be considered to be derived from practice effects that arise from repeating a task (Walker & Lindsay, 2003), IVR users also had additional attentional processing load as compared to the control group (McEwen & Dubé, 2017). We argue that any positive practice effect is likely to be moderated by the attentional processing load from IVR, thus the overall positive learning outcomes should be attributed to IVR providing a net positive experience.

It is also possible that high-performing students would show test performance gains regardless of IVR intervention. However, we note that the high-performing students attributed improved understanding to the IVR simulation, which facilitated their ability to interact virtually with the different processes and they described making connections to the concepts that they previously had not.

RQ2. Effect of learner characteristics and sensorimotor engagement on student understanding and their IVR experience

By examining students’ conceptual understanding between the control group, seated IVR and standing IVR groups, we demonstrated significantly improved outcomes for the seated IVR group compared with students who did not experience the IVR simulation (control). Moreover, for the subset of students who experienced MolGenVR, their posttest results interacted with their pretest scores, which suggest that their domain experience, as a learner characteristic, factors into students developing understanding in IVR. Diving deeper into this interaction, we found that students with low-to-intermediate prior domain knowledge performed significantly better in a seated position, while students with a high degree of prior knowledge about gene regulation did not differ in learning outcomes, whether they were seated or standing. The latter were able to practice their existing knowledge in IVR and found it helpful for making connections between discrete processes. It is possible this result may be attributed to the ceiling effect. An assessment with additional or more difficult items may discriminate differences in the seated and standing conditions for students with high incoming prior knowledge, but further study is needed.

We believe our results point to the cognitive costs related to physical, sensorimotor experiences, which may explain why students with lower prior knowledge in the seated group performed better than those in the standing group. Students have indicated a degree of comfort that allowed them to focus on the simulation content. This builds on earlier findings that suggest IVR experiences to be taxing on cognitive resources and that extraneous cognitive load in processing the visual and spatial information distracts or takes away important resources in the learning process (Makransky et al., 2018; Moreno & Mayer, 2004). This explanation works considering the results for students with low-to-intermediate prior knowledge, but not for those assessed with higher prior knowledge. It is also worth noting that, as illustrated in the IVR cases, student preference and autonomy may be influencing factors.

With our multimodal analytical approach using physiological monitoring, eye tracking and video data, we examined the IVR experience in-depth and identified elements common to both seated and standing IVR experiences. For students who were seated, interactions involving the upper body such as throwing, was limiting, as well as virtual objects “crowding” the user being a possible issue. Comfort and perceived focus are cited to be positive attributes to this mode and this specifically fills the aforementioned gap in the literature about limitations in previous studies that were not able to mitigate participant discomfort. Both the improvements in IVR technology and alterations in our study design (offering seated positions) reduced the level of discomfort experienced by participants in previous studies. Standing students were observed to move with ease, while navigating the virtual space and managing virtual objects around them. In both seated and standing cases, we believe the physiological responses associated with careful examination of the components and explanations of their function, contributed to Kara and Emily’s success in their posttests.

We provide preliminary evidence on the applicability of physiological monitoring to better contextualize learning outcomes and experience data. This fills a second gap in the previous literature which is critiqued for not being able to tease out IVR affordances from factors involved in student characteristics and conceptual understanding. In our study, we demonstrate that data from physiological monitoring can provide evidence that better connects visual and proprioceptive stimuli to IVR so that we can consider individual user characteristics as separate factors in learning. More research is needed to better understand how psychophysiological responses can support the designs of IVR applications during complex learning tasks. Last, we find that leveraging opportunities for whole-body IVR interactions have implications for embodied learning, especially for learners that demonstrate less domain understanding. As we continue to discover more refined conditions of when sensorimotor cues should be engaged, they will be important lessons for future designs of IVR in learning settings.