VR and AI Development

By Timothy C. Mack

The trouble with virtual reality (VR) is that science fiction and other popular media have so raised expectations that people are always disappointed, because they all think that fully functional VR is already here. But what VR is really good at is storytelling, simulator games, and training of all sorts.

This is to distinguished virtual from augmented reality (AR), which is also in development. However, nobody has unrealistic expectations about AR, which strongly challenges present assumptions that it will be very disruptive. There continues to be a dichotomy between physically visiting a place or event and virtually doing so. There are so many physical aspects to actual presence—smell, taste, touch—that only very limited opportunities to simulate them are now possible (such as haptic gloves). One of the most appealing aspects of real presence is serendipity: the unexpected event or outcome.

Deep human impulses are released in gaming. Many experienced players even provoke their opponents to play more emotionally and thus make mistakes. While fun can be transformational—physically, emotionally, and cognitively—there is seldom complete transfer of skills in gaming or simulations. A good example is a firefighter simulation in a burning building, which does not adequately prepare one for the heat, choking gases, and real danger. And so outcomes and success levels are different for each person.

VR simulations for training still may not prepare firefighters for the real thing. Credit:  Skeeze/Pixabay 

Clinical depression simulations have been developed to help social workers and therapists understand clients. While not always successful, these simulations do help build a community of learning between “us and them.” Different approaches to teaching work skills to a mixed group of students affect them in different ways and are successful on only a certain percentage of participants, even if the game or simulation has been well designed. It is all too easy to build bad games that work for no one, and there are lots of them around. The ideal game may be one that creates the opportunity for discussion and decision making among players during the game. And again, there is quite a wide range in cognitive/rational responses and understandings taken away from a single game by a diverse group of players.

One of the challenges in VR game design is determing what assumptions the designer can make about players' prior knowledge. Also, in the United States, there is less independent game playing; rather, gaming is usually on multiplayer online settings with a single screen for each player.

Online gaming (streaming) is becoming more of a spectator sport, but live spectator sports will endure—even broadcast spectator sports that offer no audience controls. Holographic technology is improving rapidly, however, moving toward completely immersive experiences. And this holographic view is unsually unique to each viewer, depending on where they are standing in the available viewing space, enabling multiple players/viewers to share different viewpoints and values.

Will new technologies allow for virtual visits to national parks and experiences of events that allow for greater levels of participation and observation detail? Interactive playing rather than participating in live games and events could lead to changes in attitude and even in thinking about a subject. Many games are actually on a continuum between gaming and reality.

Gaming in the workplace is growing, but it is not always digital. Games such as Escape Route (Locked Room Puzzles) now popular in employee development are often more exercises for observation, analysis, and team building. It consists of half a dozen people in a physically confined space, given clues for escape with the goal of developing a successful team approach to solve the problem. But one outcome can be a “trough of despair” where people stop responding to the game structure and innovative behavior declines. Because no one game works for everyone, it raises the questions of why people play any specific game at all. This requires understanding your community of players.

It is clear that games can communicate complex ideas to their players in ways that seem intuitive. Games can communicate meaning. New York University's Game Center and University of Southern California's Annenberg Innovation Lab are working in this arena. One important thing that games offer to their players is engagement. In order for this engagement to develop, the game must invite iteration—repetition builds engagement over time. But it is very difficult to anticipate how all players will respond to any specific game, and not everyone seeks empowerment. That lack of control can be the novel and intriguing experience. And role playing can provide all sorts of new experiences

At Google, researchers are working on natural language solutions, leading to new machine learning frameworks, including deep learning projects such as Tensorflow. The whole deep learning area is moving ahead quite quickly, as computing power advances. Besides Google, Facebook and Microsoft are committing large resources, and a number of smaller companies are also involved.

Machine-learning tools drive advances in robotic movement controls and energy management, which are two of the most difficult challenges at present. Virtual models developed to build robotic systems often were poorly conceived or even wrong—they did not work in practice. There was not enough real-world input into building those systems, and the outputs were often full of digital noise. For example, task-training data is often too scarce to inform task design.

Another real challenge is crafting strategies for interpreting emotional interaction—and reading opponents in game playing. This research is being led by Google Deep Mind (renamed after Deep Mind Technologies in UK was acquired by Google). Graphic processing units are often more effective than CPUs to communicate problem solving strategies, and the majority of present AI work relates to assisting humans rather than beating humans at games. This is not artificial but augmented intelligence.

Many are concerned about black box intelligence with full agency and independence—so some are designing and bulding attention-tracking tools allowing us to see what was incorporated in an AI decision. This will help us better understand the decision steps involved and replicate them—and help us understand mistakes, as well.

It is nearly impossible to look out 20 years in AI research because change is happening so fast. Even 10 years out is too far ahead to be accurate. But one thing that will happen is that the Internet of Things will continue to improve its understanding of users through enabled devices, as well as their desires and patterns of behavior. And deep-learning tools will inform research in areas such as biology, medicine, and energy development.

Timothy C. Mack is managing principal of AAI Foresight.

Image credit: Skeeze/Pixabay