How Disney and Marvel Designed the Vision Pro Multiverse with You as the Hero

Welcome to your multiverse.

At the end of May, I participated in an interesting experiment where Disney, Marvel, and Industrial Light & Magic teamed up to create a hybrid VR/AR experience. Immersive gaming experience Based on Apple Vision Pro What… What if? TV series.

Also: If you have an Apple Vision Pro, you absolutely must download Marvel’s ‘What…If?’ It’s free.

I consider this an experiment because it is as much a concept prototype as it is a game. First, this only applies to the $3,500 Apple Vision Pro headset. Second, The game is free. And thirdly, the game’s interface is based on the headset’s ability to recognize hand gestures. Participants move their hands in the same way as the heroes of the Marvel Universe (especially Doctor Strange) do in the movies.

As a game, it’s fun, but not very useful. The gameplay feels very linear. But the impressive and thought-provoking immersive aspects provide a glimpse into how games and UI elements could potentially evolve in a spatial computing environment.

But I wanted to know more. What were the design decisions? What were the challenges? What was Disney thinking? I mean, really. What was the team thinking?

So I asked, and Marvel Studios Director and Executive Producer Dave Bushore and ILM Immersive Associate Producer My-Linh Le were kind enough to answer.

Let’s begin.

ZDNET: What inspired you to create an immersive version of the “What…If?” series for Apple Vision Pro?

Dave Bushour: I believe that “What if…?” is the perfect entry point. It asks the audience a question from the beginning, and allows the characters and crossover points to play out in the way that best suits the story. In this case, developing the story for a completely new piece of hardware made it authentic to the series in a new way.

ZDNET: What was the biggest challenge in adapting the “What…If?” series into a mixed reality format?

Database: The biggest challenge was to start a new conversation with the audience through this new format. We knew we had to develop a story with a pacing that would truly connect with the audience, even though we didn’t know where they would look or how they would engage.

Mirinrae: We wanted the audience to feel like the main characters in the story. [and it does —DG] And how they could impact these different multiverses, for better or worse. It was a challenge to try to show as many “what if…?” worlds as possible while still keeping the audience engaged in their role as heroes.

ZDNET: How do you see mixed reality changing the entertainment industry over the next five years?

Database: Adoption. As technological boundaries continue to expand and hardware, software, and devices continue to improve, the door will open to bigger and better stories we can tell in mixed reality and what audiences want.

ML: I believe and hope that mixed reality can bring into our homes the worlds we have always wanted to visit, where we can talk and interact with our favorite characters and create our own stories and adventures with them.

ZDNET: My-Linh, that’s a really compelling mission statement. I love it! What do you see happening in mixed reality technology over the next 10 years?

ML: The next decade will be about better spatial awareness and object recognition. Once we can do that, we can make experiences more personalized and more engaging.

Database: Developing photorealistic characters in real time would be an emotional game changer. The technical challenges are so great that when that happens, the story can really be felt in people’s minds as if they lived it or are currently living it.

ZDNET: What are your thoughts on integrating mixed reality into other Marvel and Disney properties?

ML: I want to see how fans can get involved in real stories, influence the outcome, become a threat if they want, and have their own unique experiences with their favorite characters.

Database: I really think that as things evolve, there will be a universe of infinite possibilities… It will be defined by how far we can push the ideas that people can engage with. The next great story in this space is only limited by what we can imagine.

ZDNET: Can you tell us about the collaborative process between Disney, Marvel, and ILM in developing this project?

Database: Disney and Marvel have been collaborating for years on the possibilities of telling mixed reality and virtual reality stories. ILM Immersive has been a great partner and leader in this space, and with so many talented and passionate people working on this project, it has gained significant momentum both technically and creatively.

ML: With Dave guiding and leading us through his creative vision, we were always in constant communication with each other. Through story meetings, creative reviews, headset reviews, etc., we felt like a cohesive team, even though we were spread across multiple studios.

ZDNET: What are the key factors that make a mixed reality project successful in terms of audience engagement?

Database: The biggest thing that I believe is a huge factor in these types of projects overall is presence. Making the character feel real and present, and that’s something we’ve been doing successfully for years with ILM Immersive.

Also: How to use meta quests in a moving vehicle (while still in the passenger seat!)

With Apple Vision Pro, you take it to a whole new level. As a spectator, as a participant, as a hero… feeling the character’s performance and moving through the environment as if it were real creates an opportunity to create a sense of presence that you’re living in this reality. It opens up the audience’s mind to feel that it’s real, and allows them to connect with it on that level.

ML: I also think that anything that allows people to experience something new is important. If they can walk away saying they’ve never done something like that before, that’s a huge part of what makes a mixed reality project successful for the audience.

ZDNET: Well, that’s definitely the case here! How did you ensure that the immersive experience stayed true to the original “What…If?” series?

ML: We got a lot of guidance from Dave and the “What If…?” team, but a lot of people at ILM Immersive were already huge Marvel and “What If…?” fans, so a lot of them were regularly sharing their Marvel knowledge throughout development.

Database: When developing the story, our writers David Dong and Phil McCarty focused on the core properties of the Disney+ series and the core of the “What If…?” comics and the fun that those stories always have. We worked with ILM Immersive to work out some of the core ideas that we wanted to achieve creatively, applied them to the main character arcs, and then had a lot of fun saying “What if…” in every meeting for months!

Also: Are you getting motion sickness from your headset? Here’s a weird way to treat VR-induced nausea.

ZDNET: How do you balance technological innovation and storytelling in your mixed reality projects?

Database: As a director working with the writers, producers, creative team, and the Marvel Studios team, it was about what elements of the story mattered most to you as a character and as a hero, and how can we create an emotional journey for everyone? And then layering that on top of the wish fulfillment possibilities that this technology is starting to unlock. There were so many ideas that I wanted to try from an innovation standpoint, but I think it works when you combine the technology with the emotion of the whole journey that you’re trying to create.

ML: I think we generally approach it from the perspective of what can create amazing stories and experiences, and then evaluate the feasibility. We always try to push the boundaries of what technology can do and have a “let’s explore everything” mentality in every project, but above all, the story we’re telling has to be meaningful.

ZDNET: I want to highlight the phrase, “the wish-fulfillment possibilities this technology is beginning to unleash.” That’s a profound and unique phrase to this medium. What are the lessons learned from this project that will inform future mixed reality efforts?

Database: The biggest lesson I’ve learned is that the future belongs to those who can combine genre, format, development style, game features, and storytelling, because that’s where the biggest opportunities lie.

It creates unique challenges and opportunities that are both completely new and not so new for the team and the audience. The board is blank and everyone’s future projects and efforts only expand innovation and the audience’s excitement for more content.

ML: In addition, we’ve done a lot of testing and exploration for this project, and I think we’ll definitely see more in the future. There’s so much potential for what we can do.

Also: I returned my Apple Vision Pro and Meta Quest 3 for these XR glasses. They’re on sale for $140 right now.

ZDNET: How does ILM’s expertise in special effects contribute to the development of immersive experiences?

ML: Our art team is amazing. I get to work with some of the best people in the industry, and many of the people at ILM Immersive come from traditional VFX backgrounds. I am truly impressed every day by the quality and fidelity that our team can produce, and I think this experience speaks volumes about that level of work.

ZDNET: What are the biggest opportunities and challenges of mixed reality in the entertainment industry?

Database: The biggest opportunity is to create more groundbreaking creative moments, and to reposition socially through audience engagement and ecosystem expansion through hardware.

We need more genre-defining content that people can share with their friends, allowing stories to transcend the boundaries of niche entertainment and deliver undeniable experiences that define a generation for both young and old.

ML: Accessibility and reach are also major challenges in this space. Many people have yet to explore mixed reality, often due to significant financial barriers. Hopefully, over time, it will become more accessible, which will lead to a boom in mixed reality storytelling.

ZDNET: What advice would you give to other companies considering entering the mixed reality entertainment space?

ML: The best advice I can give is to push the boundaries as far as you think technology can go. You never know what’s possible until you see how far you can go.

Database: Focus on the question, “Why?” This really helped us narrow our focus and create a great sandbox where we could live out our “what if…” stories from both an innovation and narrative perspective. Why are we telling this story? Why are these characters on this journey? Why are we using this gesture, this model, this location? If it’s important to you, it’s going to be important to others.

Also: Best VR Headsets: Tested and Reviewed by Experts

Food for thought

This interview was very thought provoking for me, and I hope it is for you too. I’ve been exploring XR and spatial computing for quite some time, but when some of the greatest storytellers in history explained why it excites them, a deeper understanding took hold.

I’m excited to see what this team will explore next. What do you think? What… If? Have you tried immersive experiences? Have you jumped into VR or AR? Apple Vision Pro or Meta Quest 3 Headset? Who is your favorite Marvel character? Let us know in the comments below.

You can follow daily project updates on social media. Be sure to subscribe. My weekly update newsletterAnd follow me on Twitter/X @DavidGewirtzOn Facebook Facebook.com/DavidGewirtzOn Instagram Instagram.com/DavidGewirtzAnd on YouTube YouTube.com/DavidGewirtzTV.