Virtual reality will open the door to incredibly intimate surveillance

Maude Carrier doesn’t spend a lot of time with wild animals, so she was nervous when dropped into a temple with monkeys and baboons milling about. A few seconds in, a giant ape emerged from a dark corner and began to approach her looking surly and threatening. But her nervousness quickly dissipated when a bear intervened talking to her with the voice of Bill Murray.

“It was kind of a stressing scene because the ape is very scary but then the bear is coming, and just by his voice you can tell he’s a nice bear,” Carrier, who was visiting the scene via a virtual reality headset, said about the 360 video from The Jungle Book.

Carrier’s reaction to the video was being tracked by Retinad Analytics, which specializes in measuring people’s emotional reactions to 360 video and virtual reality. Heat maps of Carrier’s head movements were created using the gyroscopes and sensors in the VR headset. In more intensive Retinad tests, sensors track sweat, heart rate, and eye movement to see what watchers are interested in, excited by, or disgusted with. Eventually, they hope to understand exactly how the human head moves when feeling a certain emotion.

“There’s a very distinct pattern for laughter. There’s a very distinct pattern for crying, and we’re trying to crack that code right now,” said Retinad Director of Developer Relations Alexander Haque.

Oculus Rift and HTC Vive headsets only became publicly available a few months ago but there are already a handful of VR analytics startups–most no more than two years old with a dozen employees or less–figuring out how to closely track people’s emotions and interests in virtual environments. After all, analysts think that virtual and augmented reality will be a $150 billion industry by 2020.

VR analytics companies gather data about a person’s physical reaction 60 times a second. The interpretation of that data helps the creators of games and applications improve their offerings so that they can make stimulating, addictive content that will move VR toward mass adoption. The fine-tuned feedback could help filmmakers design increasingly pleasurable VR porn, allow medical professionals to create virtual treatments for people suffering PTSD, and assist psychologists to cure phobias.

At the same time, virtual reality analytics could become the most pervasive form of mass surveillance ever, allowing advertisers to mine people’s very bodies for information that could be used to manipulate them. As Wired’s Kevin Kelly put it in a recent piece about Magic Leap’s augmented reality glasses, “every virtual world is potentially a total surveillance state.”

Virtual reality analytics could become the most pervasive form of mass surveillance ever, allowing advertisers to mine people’s very bodies for information that could be used to manipulate them.

“I think if you can get into the mind of users and understand how they’re thinking about the environment, you just unlocked the whole thing,” said Retinad CEO Sam Poirier in April at the Silicon Valley VR Expo. “We don’t know how to do it yet. It’s still very tricky, but when we’re able to do this, the content in VR will probably be the best content across all media because it really will be adapted to what people are feeling.”

A focus on a user’s emotions and physical reaction isn’t just about getting inside their head; it’s about getting their minds to accept what’s happening. VR, at its core, is the manipulation of what people sometimes refer to as the “lizard brain.” To go deep into the latest technology requires tricking our prehistoric brain in order to make the virtual environment feel natural.

“It is an appeal to our older, more developed hind-brain and not our younger, more recent rational brain,” said Charles Etienne, who heads Retinad’s studies to track emotion. “To make VR feel real, we need to appeal to our most cardinal of mammalian senses. Think about that for a second. One of the greatest advances in technology is really all about massaging and coaxing the neurons in our lower-order mind.”

Audio or visual cues that feel unnatural interrupt immersion. No immersion, and a genuine emotional reaction is less likely.

“We need to consider immersion as the most essential element for VR, and we need to ensure that we are working on making the user emote to the exact same degree as they would in the natural world,” Etienne said in a recent Medium post. “From an evolutionary point of view, emotions are the best signal to identify whether somebody is feeling immersed or not.”

Once companies like Retinad identify the head movements that translate into specific emotions, creators can personalize content based on a person’s mood. If sensors detect the experience is too intense for a person, say during a VR therapy session for victims of trauma, the content can automatically dial down the experience. Or if a headset recognizes that a VR user is starting to get bored, the content can throw some stimulus in the user’s face.

“Think of that from a developer’s point of view,” Haque said. “If we can signal 10 seconds before this person is getting bored, throw a [virtual] snake at them.”

The half dozen startups in the VR analytics space interviewed for this article believe emotion tracking and physical reaction are more honest and telling than most any other kind of analytics. The different companies are trying to track emotions in different ways though. Instead of a focus on head movement, South Korea-based Binary VR uses cameras inside headsets for facial recognition. Louisiana-based Yotta Technologies uses sensors that track facial muscles and eye movements in order to capture microexpressions with the hope to track more subtle emotions.

“Everything about you is contained in a six-by-six inch space in your head,” said Yotta founder Charles Miller. “Our main goal is to unlock human emotion and have that translated onto a computer.”

“Our main goal is to unlock human emotion.”

Once they perfect emotion translation, these companies believe VR will become the most personal media ever made, an “empathy machine” that can induce any emotion and fulfill Mark Zuckerberg’s claim that VR will help us experience everything.

This is concerning to some privacy advocates. In April, Senator Al Franken, the ranking member of the Senate subcommittee on Privacy, Technology, and the Law, sent a letter to Facebook-owned Oculus Rift about its collection of data about users’ locations and movements and how it will use it.

“I believe Americans have a fundamental right to privacy, and that right includes an individual’s access to information about what data are being collected about them, how the data are being treated, and with whom the data are being shared,” he said in the letter.

In its seven-page response, Oculus said that it needs to collect physical movement information to create “a safe, comfortable and seamless VR experience” and that it needs to share that information with developers to figure out how to give people the best possible experience.

“We believe VR has the power to change the world by enabling people to experience anything, anywhere, with anyone, and know that this will only be possible if we invest in the security of our community,” said Oculus General Counsel Jordan McCollum.

Facebook, Google, and Microsoft ultimately want to combine virtual reality, big data, augmented reality, and artificial intelligence to lay information from the internet on top of the real world and create the ultimate wearable tech. Microsoft, whose Kinect camera for the Xbox already has the ability to track movement and heart rate and can quantify a range of emotions, is now working on the Hololens, an augmented reality headset. Using its Kinect tech, it could theoretically look outward at the world instead of inward at the user. At a bar, it could tell its wearer when a person is sending signals of attraction. At work, it could let you know when a co-worker is upset with you. The ability to analyze emotion in real-time is tech that could help a parent soothe a child, that could tell you if somebody really loves you, or that could help manipulate a business negotiation.

Jody Medich, who teaches courses about VR at Singularity University, believes VR and AR will reshape the physical world.

“Think about how sophisticated [artificial intelligence is] going to be able to predict, anticipate, adapt and augment humans using these kinds of analytics,” she said during a VR analytics conference in March.

What happens if a company, business, government, or loved one knows exactly what will make you laugh or smile? What you find repulsive or pleasurable? What happens when the people around you can tell exactly how you feel? We may well find out.

Khari Johnson is a reporter and nachos enthusiast living in San Francisco. He gets pretty excited about impact, tech, politics, and community.