Reimagining Language Lessons for Immersive Learning in Virtual Reality

By Amelia Ijiri, Kyoto Institute of Technology

Introduction

Virtual reality (VR) offers many attractive affordances for language learners. Virtual reality environments can simulate 360 degree views of target language environments, thus giving rise to a feeling of “being there”. Creating 3D objects from scratch or bringing them from a resource library with a click of a button for vocabulary and grammar practice brings a level of participation and interactivity to the learning environment that is impossible in a traditional classroom. With current technology, it is also possible to explore taste and smell; body perceptions like heat, cold and pressure; and tactile sensation (Piovesan et al., 2012, p. 296). Immersive experiences motivate VR users by generating a high degree of satisfaction with virtual reality lessons (Symonenko et al., 2020, p.37). For example, taking a virtual reality tour (VRT) allows students to experience what it is like walking down a city street and hearing the sights and sounds of the target language.

Students are encouraged to discover, explore, and build knowledge (Piovesan et al., 2012, p. 295). With this new technology, traditional pedagogies created for the needs of brick-and-mortar classrooms, need to be re-evaluated for affordances in virtual reality. Instructors could consider how we would structure our lessons if immersive learning was the norm? Using principles from backward design merged with practices from teaching in virtual reality, the following questions are important: What outcomes do we want from student experiences in virtual reality, and how do we get there? Why are we using VR? Why are we wearing head-mounted devices? What benefits/constraints does this have/not have? What can be achieved/accomplished in virtual spaces that cannot be done in the physical classroom? Consciously aligning learning objectives with VR technology shows potential to deliver many benefits to language learners.

Twenty-first-century skills, such as collaboration and cultural learning, are supported by VR classes (Parmaxi, 2020, p. 6). Other skills supported by VR pedagogy include fostering interaction, motivation, task engagement, vocabulary acquisition, the promotion of cultural learning and decreasing foreign language anxiety levels (Pramaxi, 2020, p. 7). Even with limitations, including technical difficulties, nausea, and lack of multimodal resources that are noted concerns, there seems to be potential for utilizing immersive environments for simulation-based language learning. This is not a new idea, but one that merits attention as we shift educational practices to digital environments as a result of the online pivot during the COVID-19 worldwide pandemic.

Some would argue that virtual reality has been available in some form since the Sensorama machine in 1962. Simulated community-building video games such as SimCity, released in 1989, never achieved long-term traction for language learning benefits, and recent immersive tools such as VR Speech, Gold Lotus, AltSpaceVR, and Virtual Speech have received mixed reviews (Symonenko et. al., 2020 p.37). However, the world has changed. The COVID-19 pandemic forced a digital shift in educational instruction. Of course, in today’s world many have grown up with digital devices and WiFi connections are ubiquitous. Today’s youth are as likely to want digital assets for their 3D worlds—for example, a skin for their Fortnite characters—as real costumes to wear. In the last five years, headsets with six degrees of freedom (6DOF), which refers to the amount of motion possible in a virtual environment, for example, the Oculus Quest 2, have made cardboard VR viewers and even Quest outdated. The new head-mounted display (HMD) retails for under 300 US dollars, and in 2021, the tech giant Facebook announced its intention to be a major player in the grab for a fully realized digital world, and popularized the phrase “Metaverse”, a term which can be used to refer to any virtual reality space. All signs are pointing toward digitized environments, not away from them.

Victory XR Academy, a Davenport, Iowa-based VR educational software and content company, is one of the companies offering high school, university, and adult classes in virtual reality as well as creating digital twin campuses. A digital twin campus is a virtual representation that serves as a real time counterpart to the physical campus. The company provides a virtual campus with immersive classrooms and over 50 courses. Students who have access to a 6DOF HMD can access recorded classes or join live classes through a subscription service. For example, Fisk University partnered with Victory XR to offer pre-med and biology students a virtual lab. 

Currently, the bulk of their VR educational products are science-based: anatomy, microbiology, and oceanography. These all work well in VR because, unlike traditional classes, for example, the 3D models of a heart can be enlarged enough for students to walk through them or small enough that they can be passed around. Dissection (human, pig, frog, etc.), underwater exploration, and room-sized DNA molecules are all perfectly suited for a VR classroom and quite difficult to manage in a traditional classroom. As Piovesan et al. (2012) emphasize, “When we can’t have real experiences, Virtual Reality is irreplaceable” (p. 298).

I was curious about teaching a language in virtual reality, and in August 2021 I received a grant from Victory XR to design and teach live five 30-minute Japanese language classes. This grant covered the purchase of a 6 DOF headset (I chose the newly released Oculus Quest 2) and classroom teaching time. VR training was provided through 15 Victory XR professional development YouTube videos which I could watch inside the headset. The school chancellor Rene, lead software engineer Danny, and Unity software developer Pateek and I had a few follow up training meetings, and Danny was online support during my live lessons and before/after classes. Working on this team was one of the highlights of my experience due to their enthusiasm. My interest was in how language learning could fit into this immersive landscape. 

In short, because language students can act and do different things in VR, the evolution of teaching and new ideas about learning must be built into lessons and conceptualized differently. This pedagogy enables more student participation and interactive learning, resulting in making learning simpler and the subject matter more interesting (Piovesan et al., 2012, p. 296). I will explain these sessions in detail, but first let’s take a look at the time and planning that went into getting started.

Project Planning 

Environments and Assets 

I was invited to zoom with Rene, Danny, and Pateek to start the conversation about the construction of custom environments and assets they could create for my Japanese language immersion class. We brainstormed what would be most conducive to Japanese language classes. The (virtual) sky was the limit, literally. While hotel lobbies, restaurants, and airports would provide good backgrounds for language practice, Victory XR wanted someplace cultural where students would be able to experience an entirely new experience, along with an imaginary classroom with cultural references from manga and anime. After some discussion about Mt. Fuji and the Shibuya crossing, Mt. Fuji was selected because of its less cluttered background and, of course, its position as an iconic image of Japan. This custom environment took about two days to complete. To bring the environment to life, a 3D asset of a flowering cherry blossom tree losing its petals in the blowing wind (sakura blizzard) was suggested. In addition, a building resembling a shrine was built and a rock garden. For the course content, the affordance of VR attempted to “represent the natural complexity of the real world by exposing learners to immersive content” (Alizadeh, 2019, p. 23-24).

Lessons that work well with virtual reality language teaching are kinesthetic and use spatial affordances to interact. Teaching with 3D objects gave me a great deal of classroom material to work with for teaching vocabulary and grammar. 3D objects are stored in a shared library. While the selection in the library was limited I was able to select from some Asian housewares, like a teapot and garden features. At the click of a button, I had 3D objects to enlarge, reduce, position, create, or delete. The students could also create hand-drawn objects using a 3D pen. All of this was in addition to conventional classroom features, like the ability to use Google Slides on a projection screen, watch 2D videos, or sit down.

HMD onboarding: Oculus Quest 2

Until now, I used browser-based virtual environments in my language classes. Known as Web VR, there are fewer equipment costs; students operate their avatars using a keyboard on desktop computers. For me, working with Victory XR was the first time conducting a class where both I and the students all wore 6 DOF HMDs for the duration of class which was fully online. There is a steep learning curve from Web VR because the digital device is physically attached to you; a headset over your eyes and both hands in controllers, meaning you couldn’t use your eye and hands for anything else while teaching. While I was training, I often wanted to look up, write, or take notes to process new information but both hands were in controllers. For example, there were about 15 ten-minute videos guiding teachers who were new to VR through the steps. They could be played on YouTube inside the headset, but taking lengthy notes on the virtual reality notepad (a feature that allows you to write by drawing letters or clicking letters on a digital keypad) was inefficient for longer instructions while inside the headset holding the controllers. I became proficient with this and other tasks with practice: teleporting (one-click navigation to a new location), walking, sitting, bringing in materials such as slides and videos, drawing in 3D, thinking about the principles of drawing a cube, or coloring in the frame of your pictures to create a 3D effect.

For me, the sequence of importing a whiteboard and loading my Google slides seemed awkward, as both of my hands were in controllers. Importing a whiteboard and loading a slide deck takes five clicks while the class waits for something to happen. While this action became smoother with practice, students are unable to see the virtual objects before they appear in the environment, leaving a little lag time while I brought some 3D objects into the environment.

Lesson Planning with Virtual Reality Affordances

While I was thinking about and planning for potential lessons in virtual reality, I returned to a story my friend told me. My friend took a crash language course during her first week in Nepal. In this course, each student was given a slip of paper and had to complete the task on the paper in the target language with locals in the environment. Her task was to go to a kiosk and buy a single cigarette. She never forgot the language for this task, having experienced the real-life context firsthand. I tried to think about how my students could also experience the language firsthand, but in a VR environment. Kinesthetic movement might be beneficial to lasting language retention, but perhaps underutilized in a university classroom setting. One of my goals was to design movement into all of the lessons. So after a review and vocabulary warm-up, each lesson tried to use a movement with the language.

These are the five lessons I taught online, while wearing a VR headset, to students who registered for the class through Victory XR and attending virtually in their headsets. 

1. Intro to Japanese: Greetings

In the Greetings lesson, we practiced teleporting, walking, sitting, and bowing (an important cultural tradition in Japan). We went through Japanese pronunciations, drew 3D objects to represent a time of day, and walked around greeting each other according to the time of day our classmate drew their object to represent. The class ended with a guided virtual reality tour (VRT) through cherry blossom trees in Tokyo’s Ueno Park. The VRT is experienced by entering a portal into a blank virtual room. Then a prerecorded 360 degree video is played, filling the entire field of vision. Unlike the virtual classroom, where students interact with objects, the video is projected around you and your avatar is unable to interact or influence the video. The videos I used for class were recorded by a Japanese YouTuber called Ofuku, who showed us a map of the walking route and then placed us into the scene by holding a 360 degree camera recording 8K high resolution video while walking through the streets recording the sights and sounds 

2. Intro to Japanese: Numbers

We practiced numbers speaking with rhythm while striking karate poses with a YouTube Video. The class ended with a guided VRT through the bamboo forest in Arashiyama, Kyoto.

3. Intro to Japanese: Writing Numbers

We practiced writing Japanese characters in the sky with our 3D pens. We jumped stones and counted aloud completing a parkour course to the sky to get a better view of our virtual Mt. Fuji (thanks to the imaginative lead software engineer who offered tech support and participated in a class so we attached an honorific san to his name, Danny-san). The class ended with a guided VRT through Fushimi Inari Shrine, Kyoto. 

4. Intro to Japanese: Days of the Week. This lesson was set at night in an urban environment. We recited the days of the week by watching a YouTube video of a YouTuber that resembles an anime character called Cyber Bunny, and then created our scene with microphones and tambourines to sing “Ue o Muite Arukō” in Japanese. This song was requested by a student, and the YouTube video was uploaded to a virtual screen on the sidewalk of the city at night. The class ended with a guided night walk VRT through the streets of Shibuya, Tokyo. The video was recorded in 2020 and included a COVID-19 protest which students found surprising, interesting, and relevant. 

5. Intro to Japanese: Survival Japanese

The last lesson was called survival because we were producing questions with the basic sentence structure and minimum key words we learned so far in an attempt to survive a language-based task in Japanese. In survival Japanese, a pack of 3D Japanese animals joined us at our glamping spot at the base of Mt. Fuji. Glamping is a combination of the words glamorous and camping, used here because our environment was an outdoor field with high-end living room sofas and armchairs positioned around a campfire. The living room furniture choice was based on the items available in the virtual asset ready-made library. We practiced grammar by constructing questions. The 3D animal assets available for the lesson on animals were part of a library for real survival preparedness, but when I brought them into our language classroom, they moved and behaved naturally, as did the crackling campfire. The students told me the whole lesson was very real, even though it was completely impossible: I was teaching this, that, and over there by suspending 3D objects in the sky for “over there”. The class ended with a guided VRT through the Gion Geisha District, Kyoto. 

Student Feedback

Researchers are studying VR in a language learning classroom’s potential to bring a whole new dimension to the realm of language education. Alizadeh’s results supports others (evidenced in the literature (Chen, Smith, York, & Mayall, 2019; Godwin-Jones, 2016; Lloyd et al., 2017; Merchant, Goetz, Cifuentes, Keeney-Kennicutt, & Davis, 2014) claiming enhanced learning and increased motivation and engagement (Alizadeh, 2019 p. 23). My students were very interested and motivated by the VR lessons, perhaps the ability to engage with the environment, or in the case of VFT. focus their attention on something seen in video. For them, being immersed in a 360 degree video with the ability to stop the video and ask appropriate and timely questions was beneficial to providing real-life authentic context to the language. We also talked about entering a convenience store together in future lessons. Having overcome the learning curve of being in virtual reality, our class was excited about the possibilities of “going together” to different places where we could practice some useful phrases while roleplaying. 

Conclusion 

Judging virtual reality as a passing phase is premature. It is still considered “an industry in its infancy” by Carl Machover, past president of the National Computer Graphics Association (Symonenko et al., 2020, p. 38). The graphics are not good enough to make it completely believable, but they are improving. For companies like Walt Disney, who shot the Lion King in 2019 using virtual reality headsets, the photorealism is remarkable. Consider if educators used that level of reality combined with a library of environments and assets for language learners – VR could offer life-like simulations for language learners that are higher quality and more varied that what is available now in textbooks, and immersive learning might have a permanent place in language education. The digital shift to an online environment could potentially create virtual exchanges between partnering institutions. 

The COVID-19 pandemic and its resulting disruption of onsite classes has accelerated the increase in online learning opportunities and digital pedagogy. Realigning target language learning for the affordances of a virtual 360 degree environment with on-demand digital 3D objects and videos offers one option that should be further researched and explored.

Bibliography

Alizadeh, M. (2019). Virtual reality in the language classroom: Theory and practice. CALL-EJ, 20(3), 21–30.

Chen, C. J. (2009). Theoretical bases for using virtual reality in education. Themes in Science and Technology Education, 2(1-2), 71-90. Retrieved from http://earthlab.uoi.gr/theste/index.php/theste/article/view/23

Parmaxi, A. (2020). Virtual reality in language learning: A systematic review and implications for research and practice. Interactive Learning Environments, 1–13.

Piovesan, S. D., Passerino, L. M., & Pereira, A. S. (n.d.). Virtual reality as a tool in education. IADIS International Conference on Cognition and Exploratory Learning in the Digital Age (CELDA 2012).

Symonenko, S. V., Zaitseva, N. V., Osadchyi, V. V., Osadcha, K. P., & Shmeltser, E. O. (n.d.). Virtual reality in foreign language training at higher educational institutions. http://ceur-ws.org/Vol-2547/paper03.pdf