Virtual production is coming of age

When film and TV production was shut down because of the coronavirus outbreak last year, companies were forced to find new technologies and locations in order to get cameras rolling again.

The demand for content grew as people were stuck inside, and the industry had to innovate in order to meet expectations.

Described as a successor to green screen, virtual production creates a 3D space, using intricate backgrounds made during the pre-production process, meaning actors can interact with their surroundings, and what is around them can be changed at the click of a button.

Today, it is extensively being used in live broadcasting and film production. The technology enlivens live broadcasts and adds incredible realism to films, all while reducing the time and effort in the production process.

Broadcasters that utilize virtual production gain an edge over the competition by not just managing data, but being able to break it down, in real time, with unique, attention-grabbing graphics. With camera-tracking technology, broadcasters are experimenting with foreground AR objects on set, which lends itself nicely to displaying the full gamut of stats as dynamic 3D objects rather than flat full-frame graphics.

In terms of revenue, the global virtual production market was valued at USD 1463.46 million in 2020 and is projected to reach USD 4733.04 million by 2028; it is expected to grow at a CAGR of 15.8 percent from 2020 to 2028, estimates Research and Markets.

Virtual production is an emerging technology that utilizes a set of software tools to integrate computer graphics and live-action footage in real time. Filmmakers and contributors across various locations can deliver feedback across digital or in physical environments, where film casts are physically working on set. Such solutions further enable filmmakers to plan and communicate their creative ideas in new and more intuitive, and enhanced ways. All these aspects would help in accelerating the growth of virtual production market globally.

The initial years of 21st century bolstered the knowledge exchange across the globe as the adoption of internet was gaining traction at individual and residential level; however, the knowledge transfer was majorly through static medium (text). Moreover, with the rising advancements in internet/telecom infrastructure (introduction of 3G and 4G), coupled with affordable prices of media devices, such as smartphones, tablets, laptops, computers, and TVs, the knowledge was majorly transferred through dynamic medium (video and moving graphics). Furthermore, owing to the affordable prices of internet, coupled with rapid advancements in camera technology, content creation gained significant traction at an individual level. Content creators across the globe are utilizing various editing and graphic tools to enhance the quality of video content to attract more viewers. Thus, with the increasing number of content creators, coupled with the growing adoption of advanced editing tools, the virtual production market is anticipated to witness significant growth during the forecast period.

Overall size of the virtual production market has been derived using both primary and secondary sources. To begin the research process, exhaustive secondary research has been conducted using internal and external sources to obtain qualitative and quantitative information related to the market. The process also serves the purpose of obtaining overview and forecast for the virtual production market with respect to all the segments pertaining to the regions. Also, multiple primary interviews have been conducted with industry participants and commentators to validate the data, as well as to gain more analytical insights into the topic. The participants involved in this process include industry experts, such as VPs, business development managers, market intelligence managers, and national sales managers, along with external consultants, such as valuation experts, research analysts and key opinion leaders specializing in the virtual production market. 360Rize; INSTA 360; Boris FX; Epic Games, Inc; Humaneyes Technologies; Adobe Inc.; Autodesk, Inc.; NVIDIA Corporation; HTC Corporation; and MO-SYS Engineering Ltd. are among the key market players.

The APAC region’s filmmaking industries are among the first to implement advanced technology and techniques in filmmaking, and the region is expected to be the fastest-growing, with a CAGR of 17.8 percent.

This is due to the increased adoption and implementation of VP in film studios in Greater Asia, India, and Australia, together with an increased demand for online video streaming and on-demand content. The growth of emerging future technologies, such as AI, VR, XR in VP bodes well for the APAC region.

With multi-sectorial applications and a high forecasted market growth rate, the up-and-coming VP market in APAC presents a lucrative opportunity for investors, SMEs, and corporates to delve into.

In addition to broadcasting, virtual production is also transforming filmmaking. The technology has come a long way since it was pioneered for on-set VFX previsualization on films like Avatar and Hugo. Nowadays, the technology can be found not just in film production but also in high-end television and advertising. It offers a real-time method of making movies and TV with actors, lighting, and VFX, all shot live in-camera.

The advances that led to virtual production’s use in filmmaking occurred because of a number of factors.

One is the incorporation of dedicated tools within the game engines from Epic Games or Unity Technologies that are geared toward virtual production, in-camera VFX and real-time previsualization as an end to itself, rather than studios having to adapt gaming technology to fit.

Another factor is the continued advancement in processing power, especially on graphics cards (GPUs), used for rendering very-high-resolution content.

The other big development has been the use of LED walls, which were notably exploited on the Disney+ streaming series The Mandalorian and Season 3 of HBO’s Westworld. Here, the output from the real-time engine, often a whole CG world, is screened live on LED walls in the studio environment matched to a camera-tracking system to enable final-pixel imagery to be captured completely in-camera.

Seen as an aid to sustainable production (there is less travel and less set building) and economical in terms of cost and time, virtual production is fast becoming an accepted way to add visual effects, and even has its own terminology; brain bar has become a commonly used term for both the line of workstations controlling the virtual production and the team running them. Output for LED walls and on-set lighting are being controlled together by industry-standard protocols, such as DMX. Remote collaboration, given a boost due to pandemic restrictions, lets multiple artists log in to the virtual production and control the lighting and add or edit the components of the CG environment on the walls in real time.

Game engines are being amped up with virtual production in mind. As well as the virtual film tools created by Digital Monarch Media (now part of Unity Technologies), MiddleVR’s plug-in enables multi-display and cluster rendering for Unity, enabling use with LED walls.

While advances in technology have made virtual production possible, virtual production is changing the workflows of the film production process.

Traditional film production comprises linear, complex processes that take up a lot of time, resources, and manpower to produce. Additionally, development efforts work in silos with incompatible resources, making collaboration a challenge. The linearity of traditional VFX development often leads to complex challenges from pre-to post-production. Cinematographers and directors have to plan scenes first, then cast the actors, and only then can post-production editing happen.

As a result, things like the fix-it-in-post mentality (where amendments are made during editing), duplicative or increased VFX artist labor, or expensive reshoots occur, leading to longer production times, thus higher costs.

Virtual production solves current issues in filmmaking by having a more iterative, non-linear, and collaborative process, and can produce high-quality imagery that is much more life-like even at the pre-production stage.

Virtual production optimizes this process by using software tools to more easily integrate computer graphics (CGs) and live-action footage in real time. As the process is not linear, collaboration is easier, so feedback from production staff regardless of location can be worked on immediately. Eventually, the technologies powering VP will result in efficiencies in time, effort, and manpower – eventually resulting in cost savings.

Conceptualization and creation of in-film assets, such as props and backgrounds, can be created during pre-production, which can be ready by the time actors are on set, saving a lot of time in the production process. Game engines, such as Unreal Engine and Unity, together with other software, work with information from devices, such as sensors and camera tracking used with actors.

Game engines provide visual development capabilities within an integrated development environment that is data-driven. Technology, such as artificial intelligence (AI) in game engines, amongst other software tools, can then render and produce the desired visual content.

Virtual reality (VR) and augmented reality (AR), together with high-powered graphic cards and game engines, allow the tech and production teams to work closely together, facilitating collaborative communication. In addition to these, upcoming advancements in AI, as well as machine learning (ML) features, are expected to be added to the mix as well, according to a market report by Grand View Research. They can then produce high-quality, near-lifelike visual content without the need for expensive assets, in a fraction of the time spent on traditional VFX productions.

Industry players are taking note of the potential of virtual production. A host of new virtual production facilities have been launched over the past few months. In India, partnerships have been forged over the past year and a half.

In Germany’s Babelsberg Studios, Arri worked with Dark Ways, Studio Babelsberg, Faber AV, and Framestore to set up Dark Bay, one of Europe’s largest permanently installed LED studios for virtual production.

Broadcast and live production services giant NEP also moved into real-time virtual production for film and TV with the recent launch of NEP Virtual Studios, following the acquisition of expert VP companies Prysm Collective, Lux Machina, and Halon Entertainment.

The UK’s largest independent virtual production facility, MARS VolumeTM opened in London in July 2021.

Samsung is supplying its state-of-the-art display technology, The Wall, to CJ ENM’s virtual studio, a part of its television- and film-production studio complex, scheduled to open in Paju, Korea, later this year. The custom virtual production volume studio will be the first in the world to leverage The Wall’s boundless LED technology, unlocking new possibilities for video content production operations and virtual production solutions. The main display will be installed in an oval shape with a diameter of 20 meters and a height of seven meters or more, creating a seemingly endless backdrop to capture content.

In India, Cineom, with the aid of Zero Density, offers a range of custom solutions for real-time 3D compositing, keying, tracking, and delivery of final pixels for projects based out of India and the Middle East. Depending upon production needs, each system includes a customised set of gears, which are tested to the finest accuracy at Cineom headquarters in Mumbai.

MediaMonks plans to launch an 8000-square-feet studio in New Delhi, with three smaller studios featuring state-of-the-art equipment and facilities to get the ball rolling with COVID-safe shoots. The studio will have features with a full sensor camera, live tracking devices, and LED panels for filmmaking with Unreal Engine.

Also in India, Bollywood studio Eros has tied up with Epic Games, developer of the Unreal Engine, to produce films using virtual technology. The partnership will deliver high-quality pre-visualisation, in-camera visual effects, and virtual production.

Radhe Shyam, a drama/historical film set to be released in 2022 is the first Indian movie to be made using virtual production. The movie stars Bahubali’s lead actor, Prabhas.

Virtual production is not just being used for films. WPP recently used drones to scan and capture (to a 15 billion point mesh) a four-mile section of forest in the Scottish Highlands. It then used the power of Microsoft Azure cloud and the Nvidia Omniverse multi-GPU real-time platform to translate this into an incredibly detailed fly-through simulation of the physical landscape on an LED wall for use in automotive advertising.

However, broadcasters and film producers are not shedding the green screen yet as there are a few hurdles to greater adoption. Like any emerging technology, education and training are key to success with virtual production. As filmmakers adjust to making more decisions upfront, blurring the lines between pre and post, directors, cinematographers, and other key creatives will begin to realize how virtual production gives them the ability to create with a much more realistic view of what the final result will look like on screen.