Meta Reality Lab sets its vision for mixed reality in the future

Meta had a tough year in 2022 as skepticism rose around its big bet on the metaverse and weakness in the core business forced it to lay off 11,000 people. But the company remains committed to its vision for mixed reality, based on a series of blog posts launched today.

Andrew “Boz” Bosworth, CTO of Meta, acknowledged in a post that bringing the vision for the future to life has been harder than the company expected.

Meta suffered a blow last week when wunderkind consultant John Carmack resigned from the company after working on its virtual reality technology since 2013. And the company has suffered a lot of ridicule from investors and others who believe it is crazy to lose around $3 billion a quarter in its Meta Reality Lab division, which makes virtual reality and augmented reality products such as the Meta Quest 2 and the Meta Quest Pro.

“Economic challenges across the world, combined with pressures on Meta’s core business, created a perfect storm of skepticism about the investments we’re making,” Bosworth said. “These are the moments that truly test people’s belief in the future. During boom times, it’s easy to make big, ambitious investments in what’s coming next. But when economic conditions turn, it’s just as easy to turn the other way: cut back on your ambitions, stick to what’s safest and most profitable today, and squeeze as much as you can from it.”

Yet Meta CEO Mark Zuckerberg has stuck to his guns by investing more heavily in mixed reality — technologies such as VR and AR — than just about any other company in the world. Bosworth said that failing to invest in the future has plagued other companies. And VR had a pretty spectacular growth year in 2022 in terms of sales of software and hardware.

“We’ve all seen the disastrous consequences of this kind of short-term thinking: hollowed out companies that gave up on innovating long ago, content to just turn the crank on an existing business until it stops working,” said Bosworth. “I grew up here and in my experience, Silicon Valley tends to reject this kind of short-termism, which is one of many reasons why America has led the world in building new technologies.”

But Bosworth said the company has been adapting — evidenced by tough calls such as the layoffs.

“But I can say with confidence that after one of the hardest years in the history of the company, Meta remains as committed to our vision for the future as we were on the day we announced it. Our convictions here are supported by a few factors that we think are misunderstood by many of our biggest skeptics,” Bosworth said.

He noted that the company reported in Q3 results this October that its daily active users on Facebook were at an all-time high, with positive engagement trends. Instagram has more than two billion monthly actives, while WhatsApp has two billion daily actives.

“We continue to direct the majority of our investments toward our family of apps as we believe strength in the core can support an ambitious agenda for the future,” he said.

He noted that Q3 results also showed about 80% of Meta’s overall investments support the core business, with the other 20% going toward Reality Labs.

“It’s a level of investment we believe makes sense for a company committed to staying at the leading edge of one of the most competitive and innovative industries on Earth,” Bosworth said. “The good news is that in the long run, we think 2022 will be remembered as a year when foundational pieces of technology enabling our vision for the future made their way into the hands of developers and users for the first time.”

He pointed to the launch of the $1,500 Meta Quest Pro mixed-reality headset in October as a key accomplishment on the road to bringing high-end VR to enterprises. The headset is packed with sensors, AR tech, eye-tracking and face-tracking tech that will be key to future applications.

“We’ll be living with the benefits of this work for decades to come,” Bosworth said. “The immediate potential of mixed reality — to see your physical environment blended with digital objects — is already being demonstrated by developers like Overlay, who built Figmin XR, an amazing app for creating virtual objects and spaces in mixed reality, giving you a glimpse of the future AR vision we’re building for.”

He said that mixed reality is about more than just displaying a live video feed of your surroundings inside the headset. For it to work properly, the headset needs to understand your room as a 3D space, recognizing the surfaces and objects around you and how they can interact with digital objects, he said.

He noted it requires technologies like spatial anchors, which allow virtual things to occupy fixed spaces in the physical world; scene understanding for reconstructing physical spaces virtually; as well as stereoscopic color passthrough, for capturing the physical world and representing it accurately in the headset with a sense of depth for greater comfort.

And he thinks we’ll use VR headsets for work, as alternatives to laptops and desktop computers. He thinks we’ll use virtual screens and interfaces at our desks or in our hands. I don’t know about that prediction, as it is pretty hard to work for more than an hour with a heavy headset on your head.

“It won’t be long before a VR headset is capable of emulating a powerful home computer setup, from a device that fits in a backpack and can be used anywhere,” Bosworth said. “The journey toward that type of device took a big step forward with Quest Pro this year. Eye and face tracking has a similar combination of possibilities, both today and long into the future. Right now, it’s delivering more expressive avatars on Meta Quest Pro, allowing people’s digital selves to more closely match their real-world facial expressions.”

Of course, those avatars don’t have legs yet. But they’re working on it.

“This is one of many steps we’re taking to create an avatar system that can improve the quality of communication, expression and connection in virtual spaces — and we’ll have plenty more to share on this in 2023,” Bosworth said. “But the longer-term possibilities unlocked by eye- and face-tracking technology go far beyond avatars. Our vision for true AR glasses will require years of progress making our devices slimmer, lighter, faster and more powerful, all while consuming way less battery power and generating much less heat.”

One major efficiency gain will come when devices are smart enough to only render the highest-resolution graphics in the small area where a user is actually looking.

“This October we demonstrated this capability for the first time thanks to the eye-tracking technology on Quest Pro, and it’s going to be driving progress in VR and AR for many years to come,” Bosworth said. “We’ve also learned that the visual quality of a VR device depends on a range of technologies that go beyond the typical metrics we’ve used when describing the displays on a computer, TV or phone.”

Along with display resolution, factors like system resolution, sharpness, contrast and color range each play an important part, he said. So does the optical stack itself, where lenses, coatings and films each make a huge difference.

“We recently shared more detail on how each of these come together in Meta’s InfiniteDisplay system for VR optics, the product of years of investment in research and development for VR and AR technologies,” he said. “That R&D will continue paying off long into the future, and earlier this year we showcased some of the research prototypes we’ve built to demonstrate these next-generation display technologies. While these early glimpses of the future of VR are important, what’s even more important is the thriving community of VR developers, creators and users that already exists today.”

He noted that game developers and users themselves have been creating apps and games that are leading to a new community of creators on the Meta platforms. This self-expression from users on places such as Horizon Worlds is thriving, he said, and Meta will help it flourish in 2023.

“VR is now at a very special moment — fundamental new pieces of the hardware and technology stack are hitting the market for the first time, and a community of developers and users is unlocking its potential in new ways, from scrappy software startups to top-tier game studios, creators and artists,” Bosworth said. “We think it’s only going to get better in 2023.”

He pointed to progress in the machine learning revolution that has been playing out over the last decade and reached a boiling point this year.

“At Meta, our biggest priorities as a company are currently benefiting from more than a decade of investment in AI,” he said.

With Reality Labs, technologies like Meta Reality or Presence Platform depend heavily on advanced AI to function.

“Meta is far from the only company working on pushing the boundaries here, and we expect to see new competitors joining us in building for AR and VR next year,” he said. “As new devices hit the market, we believe our industry will enter a new era of growth and competition that will bring enormous benefits to users and the developer community. And Meta will have new devices of its own to share, including the successor to Meta Quest 2, our all-time best-selling VR device. You’re going to love it. And you’re definitely going to love the Meta Quest Gaming Showcase in the spring, where we’ll once again show off some of the awesome new games coming in the next year.”

He said the company is finding ways to work more efficiently, but he said the vision and the long-term research effort won’t change.

“Building true AR glasses will require a massive set of breakthroughs and inventions across all sorts of areas, from lenses and miniature displays to lightweight materials and AI-powered interfaces,” he said. “While our VR devices and software get most of the public attention, we are directing about half of our Reality Labs operating expenses towards our initiatives.”

He added, “This involves one of the most ambitious R&D operations in the world today, focused on building a truly revolutionary new kind of computing platform. Whenever I visit our research labs and get to demo the early versions of these technologies, it blows my mind, and I can’t wait for the world to see the results of the things we’re building there. The VR hardware and software we’ve released so far is just the tip of the iceberg, and the best days are still ahead of us.”

Meta also gave an update on its progress in making its InfiniteDisplay and advanced VR technologies, as well as its progress on mixed reality. It noted that Meta has more than 350 patents on optical technologies and it’s going to use many of the new technologies in future headsets.