Multi-codec world is here to stay

By any measure, 2020 was a hectic year for video codecs or the compression technologies that drive streaming video. This year saw the launch of two standards-based codecs (with another due soon) from the Moving Pictures Experts Group (MPEG), the first hardware support for the Alliance for Open Media’s AV1 codec, and continued deployment of HEVC/H.265.

Video distribution has grown into a huge ecosystem, so it is only natural that new video formats arise and that companies have a variety of reasons to launch new standards. MPEG, in particular when joined by ITU-T, has been extremely successful in bringing together companies, technology, and IPR in the field. This led to success stories such as MPEG-2 and H.264/AVC, and domination in most of the traditional broadcast business.

On the internet, there has always been a proliferation of formats, where small and large tech companies played a role. Some of these formats, such as Windows Media Video 9, were standardized (as SMPTE VC-1). Other formats, such as MPEG-4 Visual, became prevalent in the (illegal) download market. In the late 90s and 00s, those co-existed with formats such as RealVideo, TrueMotion VPx, Theora, and H.263. With the introduction of adaptive streaming over the internet, AVC became the preferred format for most streaming services.

In both broadcast and internet distribution, AVC has led to a rare consolidation, and can be considered as one of the biggest technology successes in recent decades. It led to a dominance in technology rarely seen, similar to JPEG for still image coding. For a long time, the question was not, which video standard is a broadcaster using, but rather, which AVC profile are they using.

More recently there has been an increasing fragmentation again. Already from the start of HEVC standardization in 2010, it was known that significantly superior technology to AVC existed, but licensing uncertainty delayed the transition to HEVC. There is some momentum around HEVC now, primarily for premium content. But at the same time, a parallel move toward formats such as VP9 and AV1 has gained a foothold. And further efforts have led to formats such as VVC, EVC, and LCEVC.

The VVC technology race
The next-generation video codec versatile video coding (VVC) (H.266) offers significant technical advantages over its predecessors HEVC (H.265) and AVC (H.264), and competitors (example, the proprietary codecs AV1 and VP9). Finalized in October 2020, VVC achieves the same level of perceptual quality as prior video codecs with up to 50 percent improvement in video coding efficiency, supporting 4K and 8K UHD and HDR. The standard’s designers created it from the ground up with new applications in mind, hence versatile in its name.

These performance gains come at an opportune moment, as the centrality of efficient video encoding in modern life grows ever more apparent. A study by Ericsson released last year forecasts that, even as total mobile traffic worldwide is set to quadruple by 2026, the proportion of that traffic for which video accounts will grow from 66 percent to 77 percent, assuming that the industry employs efficient compression technology.

VVC was designed to meet the world’s demands for efficient video compression and support a variety of new applications. However, to unlock VVC’s potential, industry must account for thousands of SEPs. Patent holders will, in many cases, request royalties for these, and the royalty cost of implementation may play a role in the adoption rate of VVC.

Recognizing both the value of speedy uptake of VVC to worldwide commerce and development and the potential obstacle that an opaque or excessive royalty regime could pose to this goal, interested companies formed the Media Coding Industry Forum (MC-IF) in 2018. Since MPEG-LA’s MPEG-2 patent pool began in the 1990s, video codec implementers have grown accustomed to obtaining all or most of the patent rights that they need to implement a codec through pools. Three patent pools serve VVC’s predecessor HEVC, and many contend that this plurality has impeded HEVC’s use.

Accordingly, among MC-IF’s ambitions was the fostering of a single, dominant patent pool that would offer all or nearly all the patent rights that an implementer would need for a single price. As MC-IF has stated in recent press releases, the organization’s membership, which now comprises 49 companies, did not align on a single pool administrator. Two participants in MC-IF’s pool fostering process, MPEG-LA and Access Advance (formerly HEVC Advance), intend to launch separate patent pools for VVC.

With a choice of VVC pools to join, SEP owners and implementers alike are considering the depth and strength of one another’s SEP portfolios. Complicating this effort is the fact that the universe of VVC SEPs is unknowable thus far. The patent declaration data for VVC – promulgated by the standards body ITU-T and the consortium Joint Video Experts Team (JVET) – is incomplete, as ITU-T does not require SEP owners to identify specific patents as essential. Indeed, as of February 2021, just 23 companies had filed declarations of VVC-essential patents with ITU-T, which is less than half of MC-IF’s membership.

VVC fallout damages video codec adoption
The arrival of a new generation of video codecs designed for streaming-first media will disrupt the transcoding market substantially over the course of the next 10 years, according to ReThink TV. The current battle over patent pools for the next generation H.266 (VVC) codec is damaging the rollout as well.

The next generation of codecs are set to be more expensive, and this means that the total addressable market for video device royalties will grow 146 percent to USD 8.42 billion in the next decade. Meanwhile the serviceable obtainable market will grow to USD 7.62 billion in 2030. This means that patent pools will be collecting ever more royalties from the available market, with the total SOM reaching over 90 percent of the TAM by the end of the period.

The ballooning size of device royalty payments is also exemplified by the flat rate of annual video device sales, which are going to slowly taper down to 2.61 billion in 2030. Growing demand for smartphones and smart TVs will be offset by a flat market for games consoles, and shrinking demand for CTV devices, computers, tablets, STBs, and non-smart televisions.

Despite the hype surrounding the capabilities of new technologies, the rise of codecs is directly linked to adoption by OEMS, which in turn is led by consumer demand for device capabilities. This means the rise of the next generation is going to be much slower than predicted, with large changes unlikely until the latter half of the decade.

There will also be increasingly fragmented use of codecs across consumer video devices. Not only are there now two distinct camps of codec families – those descended from the MPEG lineage, and those spearheaded by AOMedia – but within each family there are multiple codecs competing for implementation by OEMs. Throughout the period, the MPEG family will cede ground to the codecs found in the AOMedia ecosystem.

Navigating a multi-codec world
New video formats pop up all the time, and companies or research institutes have varying reasons to introduce them. Some offer significant compression advantage, while others are tailored to specific use cases, yet others claim to be royalty free. Whatever the rationale behind them is, not all formats have what it takes to become successful. Some do, but they fail to create momentum around them. Successful formats tend to have some specific characteristics in common.

To move toward a new standard, a certain jump in compression efficiency is required. What perceptual quality can broadcasters offer their viewers at the same bitrate? Can they provide higher quality of experience to the viewers? Can they deliver the next step in resolution at an acceptable bitrate? Often, the question is reversed and becomes, what bitrate reduction is obtained at the same quality, and therefore, what cost savings can the new format bring.

Also read : The war of the codec

Hence the question is around a threshold that makes sense from an investment perspective. Often the 50 percent threshold is used, and this is still true for certain use cases. For the broadcast world, the cost of upgrading hardware is usually immense, and the cost/channel is an important benchmark. Fifty percent bitrate reduction at equivalent quality is probably a sane investment threshold in this world, given the large overhead in replacing systems such as head-ends. Also, here though, the industry is seeing increasing transition to software, leading to shortened adoption cycles, with a lower investment threshold.

For OTT video, it can make sense to innovate faster. There, a 20-30 percent bitrate reduction could be sufficient when trading off the costs versus benefits. In this case, the cost of encoding is usually less of a bottleneck, once reasonably fast software encoders are available. Distribution/CDN costs and Quality of Experience (QoE) will be essential here. On the one hand, bits are reduced over the network and potentially improve visual quality. On the other hand, one might get a hit in caching efficiency, client QoE, and storage costs.

What does the future hold?
With content consumption increasing due to the pandemic, most content providers are trying to reduce their streaming costs. Dropping the bitrates is one way of reducing streaming costs, but, then one cannot compromise with video quality when there are top-notch competitors in the streaming market like Netflix, Disney+Hotstar, HBO, Amazon, and Hulu.

Expectations around format adoption in the OTT ecosystem
For the current line-up of formats, the strengths are mostly present in AV1 and VVC. In different line-ups, VVC has shown to offer the best compression efficiency. On top of that, it has a large contributor base. Still, its popularity will depend, to a large extent, on its licensing terms. AV1, on the other hand, is already growing in deployment. In general, there is a serious momentum around royalty-free codecs, with a large consortium that not only dominates the internet in terms of streaming bits, but also represents the major browsers and consumer devices.

 Considering OTT streaming, AVC is still dominant, and it’s considered the safest fallback option, due to its device compatibility factor. However, one cannot negate the use of HEVC in multi-codec streaming. HEVC usage has been relatively limited, as, typically, it is considered a premium content format, including 4K and HDR/WCG content. Now coming to VP9, given its large decoder support, the codec is still relevant. Netflix, Twitch, and YouTube have already deployed VP9 leading it to occupy a reasonable share over the Internet. However, one cannot negate the value of AV1 for its encoding efficiency. In a couple of years, AV1 will become dominant for internet video traffic, and HEVC for premium content on iOS devices.

 Expectations around format adoption in the broadcast ecosystem
On the broadcast side, the situation is somewhat different. In many cases, hardware that is slower to upgrade, such as set-top boxes, has to be taken into account. In other cases, broadcasters will follow national or regional standards such as ATSC or DVB. The breakthrough of new formats will, hence, strongly be influenced by choices made in governments and standardization organizations.

Similar to 4K and other high-resolution formats being a driver for HEVC adoption, 8K could be an incentive for VVC adoption. Given the technological advancements with each passing year, the industry will move slowly toward such high resolutions, especially for sports live streaming.

How formats will be adopted depends on the use case, and a split between broadcast and OTT oriented video is expected. Nonetheless, there is growing convergence driven by a tendency toward more flexible software-based solutions in both worlds.

There seems to be industry consensus that the industry will continue to live in a multi-codec world. Although there is a lot of money involved, it is not a true war, as is often stated. Rather, this is a business reality of co-existing formats with differing use cases, and companies supporting formats for different reasons. The neutral user should not care about all these formats though, and not all formats that are being currently seen will play a significant role. Some formats will find niche use cases, while others will continue to dominate for a long time and will gradually be replaced by future successors. Others will silently fade out.

This is a situation that will live on for at least some time – although there could be ways toward consolidation again.