Xavier Delgado Anglada
Technical Product Manager, Expert in Digital Workplace and VR at NTT DATA
The arrival of ChatGPT in November 2022 marked one of those disruptive innovations that can change everything in the years to come. This groundbreaking AI model, based on language models, quickly became the center of attention due to its accessibility, ease of use, and immense potential. It opened the door to entirely new ways of interacting with technology, leading to the emergence of new professions and opportunities for organizations.
Large Language Models like GPT allow, among other things, conversations that are surprisingly similar to those we have with other people. Their ability to generate content coherently and contextually translates into the possibility of providing context to a GPT model, allowing it to respond to the user and continue the conversation smoothly and meaningfully.
This evolution has transformed how people access information and interact with technology in various fields. Furthermore, when we merge generative AI with other technologies like virtual reality, the result is an enriched, highly accessible, and more natural experience in a fully immersive environment.
In this article, we will delve into the benefits that this powerful convergence of technologies brings while examining the challenges and prospects we can anticipate in this exciting horizon.
The result of the mmixx: more realistic immersive experiences
Integrating generative AI into immersive experiences results in a key benefit: enhancement and increased naturalness. Interactions become more akin to real-world interactions, elevating the level of immersion. The use of interaction devices like keyboards or mouse in virtual reality environments is not the most comfortable option with current devices, even considering the ongoing evolution in mixed reality. In immersive experiences, voice interaction and efficient recognition become the logical choice for engaging with the experience and receiving assistance or information.
In our daily lives, we don’t simply choose options from a menu; we communicate effortlessly. The incorporation of natural language becomes a crucial element in immersive experiences, bringing us closer to experiences that mimic real-life situations.
Before the advent of language models, reliance was placed on Natural Language Processing (NLP) systems capable of identifying intentions and responding within predefined actions. However, these systems lacked the broad and adaptable knowledge demonstrated by models such as GPT. The introduction of Large Language Models (LLMs) enables us to establish specific contexts and tailor the conversation’s tone, ensuring user comfort and meaningful interactions.
In which immersive use cases does Gen AI apply?
While the convergence of these two technologies is in its early stages, and future applications are yet to be seen, we can discuss three relevant use cases that can elevate immersive experiences and are applicable today.
Artificial intelligence applied to NPCs
NPCs (Non-Player Characters) are characters that accompany users within an experience. Until now you could give them a script and they would execute it, however by connecting it to a Generative AI what we achieved is that the interaction is much more natural giving the freedom to converse with the NPC as if they were a person. If we think about the universe of training, NPCs can accompany users and answer questions about a certain training process, articulate learning objectives, or even offer complementary content. With the help of AI, we can concentrate specific knowledge and provide good context to have a natural language interaction within the VR experience.
Incorporating AI into NPCs allows us to create more personalized training contexts, adapted to the user’s knowledge level and learning pace, in less time and at a lower cost. In this way, two worlds come together, learning becomes more holistic and less scripted, which results in a better experience.
Gen AI for Digital Assistants
Another application of generative AI in immersive experiences is the case of digital assistants. We can ask these devices to play music, give us recipe instructions, or set a timer. But what if we brought these assistants into the virtual world?
The help these digital assistants could provide to users is significant, especially since immersive experiences can be complex for first-time interactions with new devices. For instance, in the context of training, these assistants could offer users relevant instructions on how to use controls, navigate through space, or perform a specific action within a complex process.
Dynamic experience generation
The union of AI and virtual reality opens the door to generating dynamic experiences—ones that are different each time they are used. How would this be achieved? AI’s ability to make decisions that change the scenario without reprogramming enables this. In a few years, we might be able to generate new experiences by simply telling an AI how we want them to be.
Challenges and horizons of generative AI
Now that we’ve discussed the main use cases resulting from the union of generative AI and virtual reality, let’s explore the challenges and next steps we can expect from this significant technological convergence.
Action as a response
One of the next steps is to ensure that NPCs not only respond to users but also take the most appropriate actions in each case. For example, in a commercial use case focused on selling items, we can expect that in the near future, we can ask AI about available colors in a model, and the AI will directly show them. In other words, we anticipate that AI will provide a coherent action based on what users want to do, demonstrating environmental knowledge rather than just textual knowledge.
More comfortable and accessible devices
Technological evolution in immersive experiences must be linked to the optimization of devices. Expectations include increasingly lightweight devices and more powerful conversations, allowing greater interaction between the user and AI. In the future, we might wear lightweight glasses, similar to those used in our daily lives, enabling us to interact with AI in a more natural and fluid manner, potentially revolutionizing the way we interact with machines.
Explainability, an unfinished subject
Explainability is one of the challenges of AI, and in the future, decisions made by AI are expected to be more transparent and understandable for users. This is a key consideration for trust and ethical decisions in AI, accelerating results and mitigating risks in areas such as health, finance, and justice. The field faces the significant challenge of addressing the complexity of models and ensuring clear interpretation to avoid biases.
Establishing ethical standards and governmental regulation is essential to promote responsible and transparent use of artificial intelligence in an ever-evolving field where explainability becomes integral to AI.
In conclusion, the combination of Generative AI in virtual reality opens a gateway to a universe of opportunities and a revolution in immersive experiences, proving that artificial intelligence is a powerful accelerator for the adoption of virtual reality. We are witnessing the beginning of a new way to relate, entertain, and collaborate in a virtual environment.
At NAKA, we have successfully implemented the integration of Generative AI for the development of immersive experiences, thus opening the door to all the benefits discussed in this article. We invite you to explore all the details of this integration, including aspects related to enterprise security, in our article NAKA Smart NPCs.
