Virtual production unlocks creativity and opportunities

In 2020, the amount of film and TV projects that were using virtual production doubled. The global film and television industries have had to adapt to survive the impact of the COVID-19 pandemic. Sets have become regimented by new safety protocols and, alongside rules requiring masks, bubbles and testing, technology has radically transformed the way productions are operating in a socially distanced world.

At a time when crew members are working from multiple locations, virtual production techniques have offered a way to overcome these physical boundaries. Cast members remain on set, performing in front of giant LED screens that play 3D immersive location imagery, and game-engine software — previously used in the development of video games — combines that live action with in-camera visual effects, augmented reality and computer graphics in real time.

Vast developments in the field of virtual production over the years have led to the introduction of solutions that allow filmmakers to transfer the motion-captured movement of an actor to virtual characters, which are then previewed in real-time, in combination with live-action footage and virtual set extension elements.

While it may seem to have arisen from immediate necessity, the embracing of virtual production techniques is an acceleration of large-scale innovations that were already poised to bring change in global production. Along with the obvious COVID-19 benefits, other positives include a reduced environmental impact and the removal of geographical restrictions to talent.

COVID-19 and new trends in safety
Prior to the pandemic some industry trends were already underway, including the growth of remote collaboration. But the onset of COVID-19 rapidly changed these from nice to haves to absolute necessities. Virtual production is next on the list.

The main forces driving the adoption of virtual production were always going to be flexibility and efficiency. COVID-19 simply added more layers, since productions now also need to think about things like reducing the number of people on site and making sure projects run safely, while still being cost effective. It’s a delicate balancing act.

Ultimately, it is impossible to say no to increased flexibility, cost savings, and creative freedom. The shift to virtual production workflows was destined to happen; the question was simply: when, and to what degree? The answer to the first part is now. The answer to the second part is still to be determined, but those who are willing to experiment and think beyond the old boundaries are the ones who will come out on top.

Increasing accessibility of game engines
The enormous impact of Unreal Engine was clear even before the pandemic hit. While the engine may have been designed for games, it allows broadcasters to do so much more on air – not just in terms of incorporating photoreal content, but in how it encourages creativity and enables teams to dream up new solutions to old problems.

Last year Epic showed off a teaser of Unreal Engine 5, which will add greater real-time capabilities and better image quality, making the real and the virtual indistinguishable. Combined with camera tracking tech like Ncam, crews can see exactly how both on-air talent and CG elements are positioned within a virtual space, allowing them to quickly iterate and adjust as needed.

This is a massive industry driver, and the Epic team has made it a focus to support these efforts including expanding their toolsets and investing heavily in the MegaGrants program, which empowers companies to create the next generation of real-time production tools. Every major graphics provider has already built their own Unreal plugin, because they know if they do not they will be left behind.

The popularity of CG, AR, and visual effects
When one looks at the top 20 highest-grossing films of the last decade, the vast majority are loaded with visual effects. It is what viewers want. But VFX is no longer limited to film blockbusters. Just look at the success that broadcasters have seen in bringing major events to life in-studio, from breathtaking storms to the recent Mars landing of Perseverance. Beyond the fact that these attention-grabbing segments look cool, visual effects can also help build more informative and relevant news broadcasts.

Studies have shown that TV viewing time continues to decline each year. But there is also an evidence that viewers would tune in more often if they knew they were going to see immersive effects on their screens. By bringing AR and CG into broadcasts, one can do everything from putting viewers in the driver’s seat at a Formula E race to taking them to the epicenter of an earthquake. Visual effects are not just for film and TV anymore, and broadcasters that do not embrace these opportunities are certain to drive their viewers elsewhere.

The rise of the virtual studio
The demand for virtual studios has increased exponentially since March 2020, with companies and broadcasters looking to shift to a virtual production in light of ongoing restrictions, according to research from transmission and production facilities search site Videstream. Historic analysis of the keyword virtual studio shows that search volume is up 90 percent YoY since March 2020.

According to the report, during the period from March to December 2019 there were a total of 69,300 searches on Google for virtual studio. In the same period for 2020, this increased 90 percent to a total of 132,300 searches. The surge in growth for virtual studios, like the pandemic, is a worldwide phenomenon, but five countries make up almost 35 percent of the demand, with searches from the United States equating to just over 11 percent of total volume.

By location
Even before everyone was talking about virtual production, digital sets were starting to dominate broadcast and live TV. Despite the prevalence of virtual sets, the industry is still in the Wild West stage of virtual production. Every broadcaster has a different workflow, so there is no one-size-fits-all solution. The key is flexibility. And as more broadcasters tune in to the power of virtual production, democratization of related tech as new tools are developed that allow teams to produce high-quality content at a faster pace.

The virtual studio system works by using a green-screen studio space, replaced in real time with a photo realistic virtual set. Productions either replace the whole set, or use the green-screen or LED walls to replace specific areas, combining physical and virtual elements. Existing sets created in any 3D modelling package can be imported into the system and a wide range of pre-made assets can be easily sourced, adapted and added into the design. Augmented reality can then add an extra twist.

Take LED screens, which have become something of a buzzword since their use on The Mandalorian. This tech allows photoreal digital content to appear while filming, dramatically reducing the need for green screens and enabling news teams to create content with better, more informed storytelling at its heart. Most organizations can easily imagine the flexibility of a set like this, but right now they are cost prohibitive for many. As the tech becomes cheaper and more accessible, there will be smaller LED setups and virtual stages that can deliver a similar level of quality, opening up new opportunities for teams of all sizes.

In terms of what this acceleration means for the industry, inevitably, the business model of pre-production, shoot and post-production is changing as broadcast technology such as VR is used more up front in the model, not just for aesthetic and practical reasons, but cost reasons also. And as they become more sophisticated, whilst keeping the costs lower, broadcaster and media companies will only become more virtual in the future, rather than less, versus live.

The race is on to be more sophisticated and as a result more real – so the industry will see advancements in things like graphics cards, like real-time raytracing to increase the quality. AI will develop also, not just in content creation for action simulation, three-dimensional texturing and post-process motion-capturing data, leading to more advanced visual effect software functions, but with virtual film or production crews being a real possibility one day too.

The demand for virtual studios has been rising in the last few years for a number of factors. With the technologies improving all of the time and the output quality of virtual engines being photorealistic, the opportunities for clients to create something special and memorable are vast. Tied in with the improvements around keying, which is the integral part of virtual solutions, virtual studios are starting to become the norm.

Outlook
Five years from now, virtual production methods, tools, and pipelines have been completely normalized across the industry. The threshold to hands-on experience with virtual production is very low today, and the tools are rapidly becoming even more accessible. At the highest end, installations that currently require a significant initial investment will also lower in price.

Production pipelines are shifting, with a large part of what is currently post-production moving into pre-production, lowering cost and re-empowering artistic collaboration across departments. Because of similar production processes, different formats and even media will partly converge, creating new opportunities for multiplatform storytelling, creative talent, and creative technology skill sets.

None of this would have been possible 18 months ago. It is fortuitous the technology is there, and the pandemic has opened people up to the possibility of doing things in a different way. The pandemic has meant that broadcasters have had to create a safer working environment for talent, and virtual studios have been able to help that with the tools that are on offer.