Deepfake of Jailed Pakistani Ex-Prime Minister Imran Khan Declares Victory in Election
Pakistan’s former Prime Minister Imran Khan declared victory in Saturday’s national election using a generative AI-produced deepfake video and synthetic voice. Khan has spent the election campaign in prison and was banned from participating in the election, but he has used synthetic media to give campaign speeches and now a victory speech by employing synthetic audio startup ElevenLabs’ technology to create a voice clone.
Synthetic Speaker
In the AI-generated video shared Saturday, Khan’s voice is heard proclaiming victory despite other parties declaring their own victory. Khan was ousted and jailed last year on charges he denies. He alleges the military orchestrated his removal, which it denies. His Ai voice clone told supporters to “defend the win.”
During the election, officials prevented Khan’s candidates from campaigning and censored coverage. In response, PTI held online rallies and used platforms like YouTube. The party turned toward generative AI late in the campaign, using synthetic voice tech to recreate Khan’s voice based on notes smuggled from jail. Khan would provide text the outline of the text, which was then fleshed out by members of his Pakistan Tehreek-e-Insaf (PTI) party into a final script. The team used archival samples of Khan’s past speeches and ElevenLab’s synthetic audio engine to create a model for Khan’s voice that could mimic the sound and rhetorical style of the former Prime Minister. Viewers were notified through captions that the message utilized his digitally recreated voice.
Deepfake Desist
Khan’s use of synthetic media for political purposes is becoming a bigger trend, particularly with tools like ElevenLabs and other voice cloning services. While Khan and New York City Mayor Eric Adams deliberately cloned their voices, Adams deploying a generative AI voice clone of himself speaking multiple languages in a set of robocalls to voters, others are using the tech for more underhanded political gambits.
Robocalls to New Hampshire voters earlier this year used ElevenLabs to make a deepfake version of President Biden in an attempt to suppress turnout in the state’s primary election. An investigation traced the calls back to a telecom provider, Lingo, which transmitted them on behalf of Life Corporation. The FCC issued a cease and desist over it, followed by an outright ban on deepfake robocalls. Deepfake technology in politics has also prompted OpenAI, Google, and other developers to come up with strategies to limit the misuse of LLMs. Most have agreed to a set of safety and responsibility principles. OpenAI even banned an indirectly officially approved custom GPT of Democratic presidential nominee Dean Phillips.
Follow @voicebotaiFollow @erichschwartz
FCC Issues Cease & Desist to Company Behind Deepfake Biden Primary Election Phone Calls
NYC Mayor Adams Uses AI Voice Clone for Multilingual Robocalls
