Jessica Brillhart is a pioneer of virtual worlds, and as the new director of the Mixed Reality Lab (MxR) at the Institute for Creative Technologies (ICT), she is bringing her unique vision to USC. Brillhart previously founded Vrai Pictures, an independent XR content studio in New York, and was the principal filmmaker for VR at Google, where she helped develop Google Jump, a live-action virtual reality ecosystem. Here, she talks about the potential of VR and challenging our innate senses.

What was the first VR or AR experience that really blew your mind?

The first experience that really got to me was Oscar Raby’s “The Assent.” It was the first experience where I felt cohesion, a sense of something truly unique. The aesthetic of “Assent” also reminded me of something I believed but had forgotten — that representation in the virtual space doesn’t have to be literal or detailed in order to be emotional and impactful. We can believe in the existence of something on the basis of a few well-thought-through details. As many artists have alluded to in the past, we can see a bull in a few painted lines.

The real kicker came when I noticed I could move toward a painting that hung on the wall. This sparked my troublemaker side. I asked Oscar, “What happens if I stick my head through this painting?” expecting him to say, like most creators often do, “Yeah, nothing happens, so please don’t do that.” Instead, Oscar responded, “Do it.”

When I stuck my head through the painting, I was pulled into a swirling vortex of color and light. The painting as a universe of sorts. Beyond being a well-placed Easter egg, it was in that vortex where I suddenly realized not just how much Oscar was aware of the potential inclinations of a visitor to his experience, but he gave a damn that I had those inclinations.

You recently joined USC as director of the Mixed Reality Lab (MxR). What’s next in the lab’s evolution?

Our goal will still be to develop technology that really pushes things forward, but I think, from my perspective, it’s also about helping to steer this ship more toward better narrative, emotional experiences that resonate on a human level and that are very effective from a functional [perspective] one as well.

Inclusive design as well as accessibility: Those are two big pillars for me and my work, and I want to bring that to the lab — to really think about making sure that we’re building this stuff for as many people as possible, not just a few. That involves collaboration on a much grander scale and not just doing something because we can, but really asking ourselves, OK, but why?

Back in the early days of cinema, people like Thomas Edison thought movies were just short novelties to watch a speeding train or a galloping horse. Similarly, many people today have a limited imagination for AR/VR. Can you paint a picture of some potential uses?

There is an unfortunate separation we tend to make where if we deem something as functional, it can’t be artful. And vice versa, if it’s art, then it can’t function as anything else. What seems to be the way forward, at least in the short term, is identifying how these technologies could help. Identify their inherent function. Why should anyone really care about VR or AR? What questions do they answer? What problems could they solve? It’s worth noting that most of these immersive technologies didn’t arise because anyone in the public sphere asked for them. More often than not they were birthed from a functional ask by a smaller group of people. The predecessor to the [Oculus] Rift that [Associate Professor] Mark Bolas built at ICT is a great example of this. In some cases, it came from a small team of engineers who just gave it a shot to see if they could build it successfully, like Google Jump’s rig in 2015.

Tell us about your work with the Golden Record.

2017 was the 40th anniversary of NASA’s Voyager mission, and consequently the Golden Record. Two probes, Voyagers I and II, were assigned to do surveys of the outer planets before traveling into interstellar space. The probes would then float on for as long as they could, taking in readings and sending them back to Earth. A copy of the Golden Record was on each of these probes, so should an alien species discover one of the Voyager probes, and provided they could figure out how to access the media, they would bear witness to a vast amount of images, music, sounds and diagrams of what it was like to be on Earth. When the probes shut down they would essentially become messages in bottles floating in the cosmic ocean.

I had the good fortune of getting to know Ann Druyan, who was the creative director on the Golden Record, among her many other accomplishments. She mentioned to me how she, Carl Sagan and the rest of the team had always felt the Golden Record was really more of a “we were here” message, a means of unifying the vast permeations of the human experience in one place.

Which got me thinking: I barely know Earth. We live in our own microcosms and barely get to visit the ones outside of our own. One of the promises of virtual reality is allowing for others to experience the world differently, through someone else’s perspectives and perceptions, so surely VR could help us reimagine the contents on the Golden Record for us aliens here on Earth, too. The hope was to do a full range of experiences that would help celebrate the Golden Record and also challenge it. I wanted to shine a critical eye on cultural representation. I wanted to connect some of the images and sounds in new ways that would help better elicit a particular human concept.

If you were a public school teacher and had access to cutting-edge VR/AR in the classroom, how might you use it?

Physics demos that allow kids to toggle various things like inertia or gravity. One step further is linking that to a historical context — what happens if we use physics as we understood it at one moment in history versus another? What would that be like? How would a ball drop or roll? There’s also chemistry experiments that would otherwise be costly or hazardous for students. Going on field trips has always been a no-brainer — taking students into the world in ways they wouldn’t be able to otherwise.

There’s a lot of promise in spatial audio experiences, so I could use audio AR to enhance stories I might be telling or lessons I might be teaching, especially historical lessons. I remember as a student watching Ken Burns’ Civil War documentaries. My history teacher had to go full stop on teaching so that we could watch it and talk about it after. Nowadays, you can have both at the same time, which, if approached elegantly, could very well extend the understanding of a particular text or lesson.

What are some rules for storytelling in VR that you wouldn’t find in 2-D videos and films?

Visitors, not viewers. Worlds, not frames. Great immersive happens when you connect visitors to worlds. It’s also not storytelling. If you’re only going for one story — your story — you’re doing it wrong.

What role do you see AI playing in future VR/AR experiences?

What’s interesting about this is thinking about the vast permeations of stories or possible narratives one can create for an immersive experience. These intelligent systems can do “randomness” in a way that we cannot ourselves create or even begin to predict. Suddenly you can create a virtual space that is aware of everything, able to react and be responsive to practically anything we choose — or not choose — to do. It can be alive and in itself an active agent. What results is, perhaps, a false sense of free will, but it’s something that will allow users to roam free and explore in a way they’ve never been able to before.

VR has been described as “one of the most powerful empathy machines.” Others bristle at this label. What’s your take?

Experience affects each of us on many levels, differently. In order to truly create empathy, one is required to consider factors that are flat out hard to quantify in any real way. Many of these effects are happening subliminally. To think that my being at a refugee camp in VR means that now I, too, know what it’s like to be a refugee is ridiculous. The concept of a “refugee” involves much more than being relocated to a camp. I think all we can hope for is that I understand that a refugee camp does exist and that there are humans in there trying to continue living. It’s the understanding that it isn’t just a photo on a page in the newspaper. This place is real. This may cause me pause. I may reflect. I may donate. I may contribute differently in conversation. As creators, that’s all we can hope for. Do I think that’s truly empathy? No. But it’s not a bad thing, either.