Zuckerberg and Fridman show off insane leap in face-to-face VR chat

“This is really the most incredible thing I’ve ever seen,” said podcast host Lex Fridman, interviewing Mark Zuckerberg in VR using Meta’s stunning new photorealistic avatars, coming soon to Quest, so lifelike that they “cross the uncanny valley.”

Mark Zuckerberg has caught a lot of snark online over the craptastic look of Meta’s earlier VR avatars – but these new ones appear to be an absolutely colossal leap forward, and a glimpse into a future in which VR communications and meetings will genuinely feel like you’re in a room with people.

Here’s how they currently work: the photorealistic avatars are currently part of a research program, and required both Zuck and Fridman to sit in a studio for hours, having their faces captured by a variety of cameras as they made a bunch of different facial expressions, which were used to build motion-controllable avatar heads.

The VR headsets in this case were Meta Quest Pro units, but the technology will work just as well on the cheaper Meta Quest 3. A number of cameras built into the headset track your facial expressions, paying particular attention to eye and mouth movements.

This movement data, plus audio from the headset’s microphones, comprises a compact data stream that can be sent cross-country with near zero latency, to drive a copy of each person’s avatar pre-downloaded to their conversation partner’s headset.

The effect clearly blew Fridman away. “The realism here is just incredible,” he said. “I think it’s the future of how humans connect to each other in a deeply meaningful way on the internet.. This is super interesting that such intimacy of conversation could be achieved remotely. I don’t do remote podcasts for this reason – but this breaks all of that.”

The current avatar capture process will soon become much simpler, taking 2-3 minutes on a smartphone. “We’re probably over-collecting expressions when we’re doing the scanning, because we haven’t figured out how much we can reduce that down to a really streamlined process and extrapolate from the scans that have already been done,” said Zuckerberg. “The goal, and we have a project working on this already, is just to do a very quick scan where you take your cell phone, kind of wave it in front of your face for a couple of minutes, say a few sentences, make a bunch of expressions.”

Demonstrating some of the world’s most advanced facial expression capture and transmission technology in an interview between two of the world’s most famously robotic humans was an interesting choice, and the irony wasn’t lost on Fridman.

“That’s gonna be the meme, that the two most monotone people are in the metaverse together,” he said. “But I think that actually makes it more difficult. The amazing thing here is the subtleties of the expression and the eyes. People say I’m monotone and emotionless, but I’m not. It’s just, maybe my expression of emotion is more subtle, usually like with the eyes. And that’s one of the things I’ve noticed, just how expressive the subtle movement of the corners of the eyes are in terms of displaying happiness, or boredom, or all that kind of stuff.”

These kinds of photorealistic avatars might be perfect for some kinds of intimate face-to-face communications, but the way they’re built, you can of course do all sorts of things with them, from talking as somebody else’s head, to exaggerating or minimizing your expressions, to ditching the photorealism altogether and driving more or less any kind of avatar.

It’s a fascinating chat, a shocking preview of the near future, and well worth an hour of your time. Check it out below!

Source: Lex Fridman