Meta Demos New Metaverse Voice Assistant and Conversational AI Platform

Meta introduced a new end-to-end neural model for voice assistants called Project CAIRaoke that it claims allows for more natural conversations with AI. The new model is already getting tested on the company’s Portal smart displays with limited functions, but CEO Mark Zuckerberg described far wider plans for the project. He also introduced a new voice assistant for the metaverse called Builder Bot that generates and sculpts the digital world.

Project CAIRaoke

Project CAIRaoke expands on the natural language processing and related research performed at Meta into a conversational AI platform for voice assistants and chatbots. The presentation showed how the voice assistant keeps track of context and ongoing events in a video showing a family making a stew and the voice assistant alerting them they had already added salt and that they were nearly out so it had ordered more. CAIRaoke grew out of Meta’s BART AI model, as well as the open-source chatbot software Blenderbot that Meta has been developing over the last couple of years. A basic version of CAIRaoke is already used in Portal smart displays, but Meta sees augmented and virtual reality equipment connected to the metaverse as the ultimate home for CAIRaoke assistants.

“With models created with Project CAIRaoke, people will be able to talk naturally with their conversational assistants, so they can refer back to something from earlier in a conversation, change topics altogether, or mention things that rely on understanding complex, nuanced context. They will also be able to interact with them in new ways, such as by using gestures,” Meta AI senior research manager Alborz Geramifard explained. “We believe that the progress made with Project CAIRaoke will enable us to deliver richer communication between people and AI that will be an essential tool as we build for the metaverse. A Project CAIRaoke-powered assistant built into AR glasses could one day follow along in many new and useful ways.”

Metaverse Builder

The intersection of voice AI and the metaverse served as the foundation for another new tool demoed by Zuckerberg called Builder Bot. Through voice commands, Builder Bot is able to create and populate digital worlds in the metaverse. Though he referred to Builder Bot as an “exploratory concept,” it was able to produce a landscape of grass and trees when Zuckerberg said “let’s go to a park,” and then change it to sand and water when he followed up by saying “actually, let’s go to the beach.”  You can see how he ran the bot through its paces by asking it to make and change clouds, design an island, and even set appropriate background noises. The 3D models are relatively simple compared to what advanced software engines like Unreal can create, but the specificity of the controls and how easy they are for non-programmers to use is notable and will likely draw plenty of attention as and when it is made publically available.

Follow @voicebotai   Follow @erichschwartz

Disney Appoints Metaverse Chief to Blend Physical and Digital Worlds

Facebook Removes “Hey Facebook” Wake Word on Portal Smart Displays

Meta and Ray-Ban Stories Smart Glasses ‘Hey Facebook’ Voice Assistant Adds Messenger Controls