How YouTube Star Taryn Southern Produced A Virtual Human Clone of Herself

Musician and artist Taryn Southern’s face and voice have been seen and heard by millions of people since rising to fame as a YouTube star in 2007. Now, she’s partnered with video tech developer Hour One to generate a digital clone of herself. The synthetic voice and visuals combine to produce the virtual human version of Taryn, a photorealistic avatar that can be programmed to host a new video series she is developing.

Virtual Taryn

Southern’s digital clone, as seen in the video above, took a relatively short time to produce. Hour One filmed Southern in front of a green screen from many angles to be able to generate her face and recorded her reading several scripts and performing a song to capture her voice. That was all the material the platform needed to start transforming scripts into a huge variety of videos seemingly hosted by Southern but actually handled by her digital twin.

“From a production and performance perspective, the process was quite simple. Seven minutes in front of a green screen,” Southern told Voicebot in an interview. “As the technology improves, I could see there being a lot more performance requirements to capture a wider range of emotion for the algorithms. But, for now, it’s relatively straightforward. The result is uncanny. Not 100% realistic – you can certainly spot the differences – but this technology is impressive.”

Digital clone is the term used by Hour One but is slightly off as the avatar’s AI is solely for translating a script and stage directions into a realistic video of Southern. There’s no flexibility to the responses the way there is with Alexa or Google Assistant. The tech is impressive for what it can do, but the videos are more like digital puppet shows than interactive AI assistants. Platforms for building and publishing virtual humans have been popping up more and more of late. Digital avatars for videos are getting better and easier to make, with companies like Hour One pursuing a piece of that market. Synthesia has built a broadly similar platform, but the startup, which recently raised $12.5 million, is aiming more for the marketing and educational video market. It has run plenty of celebrity campaigns with synthetic celebrities, however, including David Beckham, Lionel Messi, Snoop Dogg, and Santa Claus.

“The technology opens up a whole world of possibilities. Not for all of entertainment, but for certain formats, like those that require hosts in a studio, for instance. For my own work, I think right now experimentation is key to figuring out the best use cases,” Southern said. “The series I’m developing, Is This For Real, will really explore the interplay between AI clones and their real-life counterparts from an entertainment perspective. I’m excited about that.”

Virtual Humanity

Southern’s digital clone shares some similarities with the virtual influencers built by CoCo Hub, including a virtual influencer AI who writes pop songs. The AI is also what makes CoCoHub’s creations interactive, which the digital Southern is not as of yet. That’s what people most commonly think of when referring to digital humans. The interactive AI can mimic real people too, like the recently unveiled Einstein digital clone, or purely fictional creations like the Nestle Toll House “cookie coach” Ruth, or Virti’s doctor-training virtual humans. Not every company in the space is going for visual and speech at the same time, too. MetaHuman Creator’s ultrarealistic visuals rely on voiceover work still, while Replica Studios sticks to generating voices with AI, although with an eye toward making it easier to integrate them with digital video. Overall, the pace of innovation is accelerating, which Southern sees as the opening of a whole new frontier in storytelling.

“I think as tech develops, people will develop more of a meta-awareness around AI voices and synthetic characters,” Southern said. “AI tools are opening a playground and giving an opportunity to be scrappy and create something brand new. I imagine AI will highlight a whole new crop of storytellers.”

Follow @voicebotai   Follow @erichschwartz