CLICK this icon for video


Accelerating the animation pipeline without increasing the amount of kit required is the principal aim of the Smart Remote Production For Real-Time Animation Accelerator project, which is bringing together leading broadcasters and vendors to leverage the latest technological developments of markerless motion capture and speech-driven facial animation to drive CG performances in the Unreal real-time render engine. “We don’t usually work directly with vendors, so this is very interesting to us,” says RAI’s Roberto lacoviello. Indeed, Italy’s RAI is just one of a strong broadcast contingent of Accelerator Champions that also includes RTÉ, VRT, YLE and the EBU, working alongside the Entertainment Technology Center at USC, Digital Domain, and Unreal Engine developer Epic Games. With participants including RADiCAL and Respeecher, with guidance from IBC Accelerator supporter Nvidia, this Accelerator represents a powerful conglomeration of talent and expertise that is looking to create the most effective, low-cost pipeline that they can to take material from script to 3D character in a real-time, distributed workfl ow environment.


One of the key elements to this Accelerator is the phrase ‘low-cost’. As well as testing the feasibility of using vocal performance and body posture to drive 3D avatars from remotely connected locations, its stated aim is to do all this while using minimal equipment. Even more so, according to one of the project leads, RTÉ’s Ultan Courtney, the wish is to utilise technology that exists less in production facilities and more in the pockets of the people that work there and their audiences. “As opposed to having studios doing volumetric capture and other high-end techniques, we want to look at a workfl ow that engages millions of people,” he says. “Most households have a smartphone, for example, so we’re working on that basis and what is the most accessible, highest tech we can work with to tell stories and communicate.”

This changes the technological emphasis of performance capture markedly. As opposed to a high technology solution involving wearing marker-based motion capture suits in a precisely delineated and monitored capture volume, the tech emphasis is shifted to the AI processing of a standard video signal. The AI tracks the performance and converts that into an avatar, mapping limb and facial movements onto a virtual character.

“This is the difference between now and a few years ago; there is a lot of AI technology that can help us at a low budget,” explains Iacoviello. “There are three aspects where AI has made a lot of progress: marker-less capture, emotional response and speech animation.”

VOICE CLONING Speech is one of the areas where this Accelerator is using interesting new tools. Respeecher is software that

Respeecher software has already been used on The Mandalorian to create the voice of a young Luke Skywalker

“I was really happy to see so many broadcasters involved in this project.

What I really hope already is that we keep going on this project even after IBC is over. We are learning so many things,” Paola Sunna, EBU

Smart Remote Production For

Real-Time Animation Champions: RTÉ, EBU, RAI, VRT, YLE, ETC/USC, Digital Domain, Unreal/Epic Games Participants: Respeecher, RADiCAL

effectively clones voices using Deep Learning techniques and has already been used on The Mandalorian to create the voice of a young Luke Skywalker and by the NFL to recreate the voice of deceased football coach Vince Lombardi at this year’s Super Bowl. “In this project the idea is to make one source speaker be able to speak in several different voices for the POC piece,” says company co-founder Alex Serdiuk. “To date we’ve offered a white glove service where we need to be involved, but this uses our new self-serve Voice Marketplace. There are many challenges for us with this as we lose a lot of control over the recording conditions and performance levels of the actors involved, but it is important to be able to democratise these tools so that they can be used in the sort of environments that ideally would require just an iPhone.”

Of course, one of the challenges any Accelerator team faces is that while the tools may be available to make the POC a success, getting them to work together in a designated workfl ow is not always easy. Sometimes they are lucky and the work has already been done. For instance, once motion has been captured using RADiCAL and uploaded to the cloud, RADiCAL provides an animated FBX fi le which can be used in both Omniverse and Unreal to retarget the animation to CG characters.

In this case, as an illustration, the wrnch CaptureStream iOS-based AI capture software can already export into Nvidia’s Omniverse real-time simulation and collaboration platform via the wrnch AI Pose Estimator. However, other areas of the workfl ow still need work. Despite Omniverse’s success at becoming a universal solvent for 3D tools, there are always challenges. For example, can the team’s Pose Estimation workfl ow be easily connected to the speech-to-ßfacial animation workfl ow that the team is working on? And can the resulting characters then be imported from Omniverse and integrated into Unreal environments? “We are pushing things forward but not everything is designed to be opened up to other uses. There are a few steps forward and back, and that’s where the Accelerator helps; having a target forces us to crash through all these problems,” says Courtney.


As the RAI team, which also includes Alberto Ciprian and Davide Zappia, points out, the TV market is not the only one that will be interested in the concept of realistic real- time character animation. Virtual infl uencers, for example, become a distinct possibility, while its use in high-end pre-production or even to drive real-time performances in mixed reality LED capture volumes is fairly compelling. “When you go into the digital world the use cases are limited only by your imagination,” says Zappia. Paola Sunna, senior project manager at EBU Technology & Innovation, says: “I was really happy to see so many broadcasters involved in this project. What I really hope already is that we keep going on this project even after IBC is over. We are learning so many things.”

For more information on the IBC Accelerator Media Innovation Programme, supported by Nvidia, visit ibc-accelerator-media-innovation-programme

Page 1  |  Page 2  |  Page 3  |  Page 4  |  Page 5  |  Page 6  |  Page 7  |  Page 8  |  Page 9  |  Page 10  |  Page 11  |  Page 12  |  Page 13  |  Page 14  |  Page 15  |  Page 16  |  Page 17  |  Page 18  |  Page 19  |  Page 20  |  Page 21  |  Page 22  |  Page 23  |  Page 24  |  Page 25  |  Page 26  |  Page 27  |  Page 28  |  Page 29  |  Page 30  |  Page 31  |  Page 32  |  Page 33  |  Page 34  |  Page 35  |  Page 36  |  Page 37  |  Page 38  |  Page 39  |  Page 40  |  Page 41  |  Page 42  |  Page 43  |  Page 44  |  Page 45  |  Page 46  |  Page 47  |  Page 48  |  Page 49  |  Page 50  |  Page 51  |  Page 52  |  Page 53  |  Page 54  |  Page 55  |  Page 56  |  Page 57  |  Page 58  |  Page 59  |  Page 60  |  Page 61  |  Page 62  |  Page 63  |  Page 64  |  Page 65  |  Page 66  |  Page 67  |  Page 68  |  Page 69  |  Page 70  |  Page 71  |  Page 72