Meta’s next big bet for VR: your living room

Forget magical virtual worlds. In its quest to broaden the audience for virtual reality, Meta is now embracing much more familiar surroundings: Owners of Meta’s Quest VR headsets will soon be able to create digital replicas of any room in their house, and then invite others to “visit” them in those spaces. 

Imagine, for instance, having a spontaneous family reunion in a metaverse version of your living room – perhaps even with an avatar that looks just like you, and not a character that has escaped from a video game.

“There is something very magical about scanning a space that you know, bringing someone else who knows that space into it and feeling like you’re there together,” says Vishal Shah, the vice president of Meta’s metaverse.

That magic, in turn, could help Meta turn its vision of a 3D metaverse as a social-3D realm into a reality – one that has cost the company close to $70 billion to date.

When your headset is also a camera

Meta demonstrated the first version of such digital replicas with an app called Hyperscape at its Connect developer conference last year. In the most recent version of the app, people can explore high-resolution 3D captures of a handful of places, including celebrity chef Gordon Ramsay’s home kitchen and Chance the Rapper’s recording studio.

The scans look so detailed and real that you can feel your mouth water when inspecting the ham on Ramsay’s kitchen counter. Meta even felt the need to add a warning about not leaning on any of the furniture in these virtual rooms.

Chance The Rapper’s recording studio [Animation: Meta]

But Meta’s Hyperscape ambitions don’t stop there: With an impending operating system update, Meta Quest 3 and Quest 3s owners will also gain the ability to scan their own rooms with their headset’s built-in cameras. “My first thought was that they probably took a very expensive camera rig to capture these data sets because they look really quite lifelike,” says 3D capture tech expert Michael Rubloff. “That all of the scenes were captured with just a Quest device [is] completely mind-blowing.”

Capturing a room with a Quest VR headset is a relatively simple process. First, the headset overlays everything in the room with a kind of mesh of geographic shapes to record its general dimensions and the rough outlines of furniture and other objects. In a second pass, it fills in those shapes with 3D data, a process that to the naked eye looks like generating a mosaic of lots of little photos. Finally, the headset prompts people to look up and around to capture additional height information for any given room.

Gordon Ramsay’s home kitchen [Animation: Meta]

The whole process of capturing an average room takes less than 20 minutes, according to Meta employees who worked on the project. Then, the raw capture data gets uploaded to Meta’s servers, where the 3D replica of the room gets rendered over a couple of hours. Once ready, each space will be streamed directly from the cloud — no time-consuming downloads required.

Meta’s digital-room replicas are powered by a novel technology known as Gaussian splatting. In a nutshell, Gaussian splatting doesn’t just capture the surfaces of objects like a regular photo camera would. Instead, it deconstructs every object into a collection of three-dimensional blobs, complete with information on how those blobs look from different angles, along with attributes like transparency. 

To date, most Gaussian splats have been captured with cell phones. However, turning the VR headset itself into a capture device has some distinct advantages. For one thing, Meta controls the hardware, which allows the company to optimize its code for a certain type of camera, instead of having to work with a myriad of different smart phones. Plus, people tend to wave their hands too quickly when trying to capture something. “The head movement is not as fast as the phone,” explains Meta research scientist Jan-Michael Frahm.

Next step: adding avatars

At launch, spaces replicated with Meta’s Hyperscape will be private, and only available to the person who captured them. The company is working towards letting people share their captures, and eventually turn them into locations for social get-togethers. 

Hyperscape captures already run on a game engine that Meta is using for its Horizon Worlds metaverse. Currently, Horizon Worlds is essentially a collection of games and spaces generated from computer graphics that people can explore together in VR. In the future, Horizon users will be able to import their own Hyperscape rooms into Horizon, and invite their friends to join them on a digital replica of their living room couch.

“I think there’s a real human connection opportunity here, where the environment is just as important in some cases as the people,” Shah says.

It’s also an opportunity for Meta to expand VR beyond its current audience. The company has had more success than some critics give it credit for in establishing VR as a medium for video games and adjacent experiences, including gamified workouts. Meta had sold close to 20 million headset sales in early 2023, and some developers have been able to turn games for Meta’s Quest headset into real money makers. Ten apps on Meta’s Horizon store have generated more than $50 million in revenue, while the number of apps with more than $1 million has surpassed 300, according to data shared last month by Meta executives.

Mademoiselle Collette French Bakery [Screenshot: Meta]

But recently, Quest headset sales seem to have plateaued. Some developers have also complained about declining revenue amid an influx of younger users primarily interested in free titles like the hit VR game Gorilla Tag. Meta aims to counter those trends by broadening the appeal of VR among older users who may not be as interested in gaming. This includes a greater emphasis on traditional entertainment, including a partnership with James Cameron to produce 3D content for Quest headsets.

That move mirrors efforts Apple has taken to promote its Vision Pro headset, which has faced its own set of obstacles. Priced far above Meta’s hardware, the Vision Pro has seen tepid sales, despite integrating with the company’s computers for professional use cases. 

But that has done little to slow a broader industry interest in VR headsets and 3D technologies: Samsung and Google are expected to launch their own headset, code-named Project Moohan, later this month. Like the Vision Pro, it is geared towards immersive entertainment and work use cases.

The company has also been working on more lifelike representations of users in VR through 3D-captured personas the company calls codec avatars. While still in development, Shah believes codec avatars could be the perfect complement for Hyperscape. “You’re in an environment that looks photoreal. You are with people who look photoreal,” he explains. “For some people, that’s going to be the most magical thing in a headset.”

Even without those avatars, 3D capture could become an important time capsule for consumers. “The same way photography has aided us with memory preservation, 3D also fulfills that promise for general consumers,” argues Rubloff. “It gives us the ability to really step back into a moment in time. “[We’ve been able to] capture the world in 2D for the last 200 years.  [Now, we’re] able to do the same in 3D.”