ElevenLabs Releases Generative AI Voice Translation and Dubbing Tool
Synthetic voice AI startup ElevenLab has released its AI Dubbing tool for translating speech into different languages while retaining each speaker’s distinctive voice. The new feature leverages the startup’s Eleven Multilingual V2 generative AI model to replicate a speaker’s voice in many languages for streaming videos, video games, and other content.
ElevenLabs Speaks
ElevenLabs pitches AI Dubbing as much more than the standard system of translation or subtitling because it preserves the vocal identity and delivery style of the original speaker. An actor’s dialogue would theoretically maintain the emotion, cadence, and other non-textual elements in more than 20 languages, delivering greater authenticity than might be possible by simply replacing the actor’s audio with someone else. The AI detects speakers and filters out background noise before cloning their voice and rerecording the dialogue in whichever available languages the user wishes.
“The release of AI Dubbing is our biggest step yet towards eliminating the linguistic barriers of content. It will help audiences enjoy any content they want, regardless of the language they speak,” explained ElevenLabs CEO and co-founder Mati Staniszewski. “And it will mean content creators can easily and authentically access a far bigger audience across the world. Having experienced these language barriers first hand, my co-founder Piotr and I are so excited to finally bring AI dubbing to life.”
As only 6% of the world’s population are native English speakers, ElevenLabs hopes creators will employ AI Dubbing to vastly expand the potential global reach of their content. You can see how it works in a long-form version below, with the audio tracks on YouTube provided by ElevenLabs and its AI Dubbing.
Generating Babel
ElevenLabs released Eleven Multilingual V2 out of beta earlier this year as a foundational AI model to build on its earlier generative AI voice models created from scratch or cloned from existing voices. The model came in the wake of a $19 million funding round in June and the release of a tool for detecting the sound of its synthetic voice models. ElevenLabs has already partnered with companies like generative AI video startup D-ID, audiobook publisher Storytel, and game developer Paradox Interactive to boost its presence in the increasingly popular facet of AI translation and dubbing. Startups like Veritone, Papercup, and DeepDub are all racing to capture some form of that market. The tech giants aren’t ignoring the space either. Meta’s Voicebox model for voice clones and its SeamlessM4T ‘universal translator’ might end up pursuing some of the same clients as ElevenLabs.
Follow @voicebotaiFollow @erichschwartz
Generative AI Speech Startup ElevenLabs Raises $19M and Releases Deepfake Voice Detector
