Say goodbye to subtitles on OTT with lip-sync tech

In these days of OTT platforms, everyone is watching movies from across the world, thanks to subtitles. But for many, by the time they are through with reading the subtitles, the action on the screen would have taken a crucial twist. And that is where a deep tech Artificial Intelligence web app built by two PhD scholars of the International Institute of Information Technology-Hyderabad’s Centre for Visual Information Technology is poised to make a difference.

The patented software from a startup, NeuralSync AI, mentored by IIIT-H, can morph authentic-looking speeches with a single video clip of a human or digital avatar, in a matter of minutes, in the language of your choice. “If it is integrated into, say Netflix, the subscriber can watch a Spanish movie sitting in Telangana, with perfectly dubbed dialogues in Telugu,” say the brains behind the concept.

According to an IIIT-H blog by Deepa Shailendra, NeuralSync’s Way2Lip is a compact video editor with an AI algorithm that is trained to generate accurate lip movement based on the target’s speech.

The project is already a hit, according to NeuralSync’s software developers Rudrabha Mukhopadhyay and Prajwal KR and CEO Pavan Reddy, who say they are getting many enquiries worth several thousands of dollars already. The US-patented technology owned by IIIT-H was invented by Rudrabha, Prajwal, Prof CV Jawahar and Prof Vinay Namboodiri.

The software and its web app slash the dubbing process to minutes. “To create one minute of video today, it will take 45 seconds and we hope to bring that down to 15-20 seconds,” says Rudrabha, adding that it cuts operating costs substantially. The technology is such that video segments can be shot with a smartphone. Rudrabha adds that almost all European and Asian languages, except probably Mandarin can be dubbed, and more importantly, with AI entering video production, mismatched lip movements will be a thing of the past.

The startup is targeting ed-tech companies, entities that create e-learning lectures and marketing video films. They are also in talks with a few metaverse companies in China. “The movie industry is an end goal where typically, we will create our product in a small Google Chrome extension,” says Rudrabha, adding that for the movie industry, a full-fledged version is in the works. The startup is also in talks with a Bollywood movie producer who has signed up a contract for executing a few scenes. Telangana Today