The war of the codec

Video is taking over the world. It is projected to account for 82 percent of internet traffic by 2022. And what started as an analog electronic medium for moving visuals has transformed into a digital format, viewed on social media platforms, video sharing websites, and streaming services.

As video evolves, so does the video encoding process. The way video is consumed is also evolving, moving from SD to HD, and eventually from 4K to 8K. To keep up with the changing video landscape, the broadcast industry has had no choice but to explore new video-compression techniques. Consequently, the industry has begun looking to new solutions to produce further advances in compression.

Part of this evolution involves developing new codecs – encoders to compress videos plus decoders to decompress them for playback – to support higher resolutions, modern formats, and new applications such as 360-degree videos and virtual reality.

There are currently two mainstream codecs: MPEG-2, the historic codec used for SD and initially the first digital deployment; and H.264, otherwise known as advanced video coding (AVC), which was established for the transition to HD. These standards-based codecs have proved to be the two most successful so far, and were both primarily formulated for the broadcast market. ITU standards like AVC were designed with TV in mind, and have subsequently been extended to over-the-top (OTT).

As the broadcast industry develops new standards to handle new resolutions and more sophisticated content types, codecs are set to change even further to support the rise of OTT consumption, and the resulting advancements in technology. These new codecs are likely to shake up the market in 2020 and beyond, and bring with them a degree of complexity, particularly as each of them is due to be set in stone over the course of the next year. As a result, it may take until next year, or even 2022, before it is possible to determine whether they will find their market.

High-efficiency video coding (H.265/HEVC) is in the process of replacing H.264/AVC, yet codec war between AOMedia Video 1 (AV1) and HEVC is already waged on. Whether a broadcaster or pay-TV operator uses AV1 or HEVC depends on their need and existing infrastructure as there are similarities between the two.

Currently, AV1 is seen as the codec of choice for streaming media distribution, and is supported by the likes of Amazon, Google, and Netflix. The next-generation codecs – versatile video coding (VVC), low-complexity enhancement video coding (LCEVC), content-adaptive encoding (CAE), and MPEG-5 essential video coding (EVC) – are also emerging options, upending the war of the codec.

Bandwidth-efficient HEVC
Since standardization in 2013 HEVC/H.265 has entered widespread tests and deployments across a whole spectrum of applications from satellite DTH, contribution, OTT, and VoD to 4K UHD broadcasts, where bandwidth efficiency is required alongside high picture quality.

HEVC is the first codec wholly designed in the modern video environment, where file movement and delivery to the home is becoming the standard. It offers up to 50 percent bandwidth efficiency over H.264 for supporting twice the number of SD and HD channels over the same bandwidth without decreasing the quality of experience delivered to viewers.

This frees up bandwidth for new services like 4K UHD as well as expanded delivery options, such as OTT and LTE-enhanced mobile video and, in certain countries and regions, DVB-T2. In addition to increased bandwidth efficiency, HEVC also supports picture-quality improvement like enhanced color gamut (BT.2020), higher bit depths (10-bit and beyond), high frame rates (50/60/100/120), and HDR.

Although increased revenue from consumers is the most widely publicized goal, there are equally and potentially more immediately addressable applications of HEVC in the file-based operations, on which media companies rely. HEVC is not simply a benefit for last-mile delivery; it has important and pressing advantages in the back-end movement of video.

One example can be seen in the flow of moving video material between an organization’s facilities. In a recent report, the average cost of a satellite transponder (worldwide) was stated to be USD 1.62 million per year for 36 megahertz of capacity. At that price, it is incumbent on operators to make the most of their existing bandwidth.

The ability to get more from a single transponder translates into a significant operational cost saving. So, when a network production center in Los Angeles needs to send material over a satellite to a distribution center in New York, HEVC provides the advantages of either moving better-quality video (in the same transponder bandwidth), more video, or simply the same video at a lower bitrate.

Additional savings may be found in storage capacity for VoD and web distribution. HEVC provides the benefits of storing content in less space than current formats and/or storing higher-quality content in storage capacity that is equivalent to what is used today. And while the benefits of monetized VoD services are predicated on consumer-level devices being able to take that HEVC stream and decode it, such devices are already coming to the market.

Royalty-free AV1
Initial demonstrations of AV1 have shown improved quality and bitrates. Companies like Netflix have expressed the need for AV1 to show a 20-percent efficiency improvement over HEVC, when measured across a diverse set of content, and would consider a 3x to 5x increase in computational complexity reasonable. The potential in bandwidth savings for content providers from using this technology is enormous.

Bitmovin points to the codec’s capability in a number of areas. These include AV1’s film-grain synthesis, where the goal is to de-noise the initial content before encoding it and then add back the noise or grain effect before output during the decoding process.

This way, the unnecessary information would not have to be transmitted at all, and the overall load of data could be reduced substantially.

Filtering is an essential process in every video codec, as it drastically increases the perceived quality of the encoded video. It mostly occurs along the outlines of each of the blocks, which are used to divide each picture into smaller sub-units during the compression process. AV1 contains various sets of filters, most of which are derived from the existing codecs.

AV1 is also said to mark the first time that non-planar motion compensation has been implemented into a video codec. Motion-compensation algorithms have been used and theorized upon for a while, but only on a two-dimensional level. AV1 changes that.

Due to the constant increase in processing power of consumer devices, this technique is now ready to see its use in mass-market applications. These techniques work extremely well for predicting large area movements, like background motion or camera movements.

Additionally, they can handle consistent backgrounds and color schemes very effectively, which is one of the reasons why animated videos tend to deliver great encoding results, even with very high levels of compression.

AV1 is also claimed to be better at handling 8K UHD because the increase in the size of individual coding units (block size) is considered a more effective way to scale the compression process, along with high resolution content, than using smaller blocks.

Perhaps AV1’s most important feature is not a technological one: it was designed from the very start to be completely royalty-free, in an effort to provide a truly open video codec, capable of providing high-quality video streaming at lower bitrates.

With the availability of high-resolution content constantly increasing, and technologies like VR and 360-degree video on the rise, the need for a suitable, technologically advanced and open codec has become apparent among large-scale content providers.

This desire is probably best documented by the fact that virtually all leading industry players and tech companies are contributing members of the Alliance for Open Media (AOM). These include Apple, ARM, Cisco, Facebook, Google, IBM, Intel, Microsoft, Mozilla, Netflix, and NVIDIA.

According to Mozilla, it is meant to replace AVC as the predominant video format for the web, and to compete with the HEVC codec, so high-quality video can be shared freely and efficiently on the open web platform. In terms of performance, tests have shown between 30 and 40 percent better compression than AVC and HEVC, but lower encoding speed.

Netflix has started to stream titles in AV1 on Android in what could significantly help the 2-year-old media codec gain wider adoption. By switching from Google’s VP9 – which it previously used on Android – to AV1, its compression efficiency has gone up by 20 percent. For now, only select titles are available to stream in AV1 for customers, who wish to reduce their cellular data usage by enabling the Save Data feature.

Netflix eventually wants to run out AV1 to all its platforms. The company is working with device and chipset partners on extending the codec into hardware. It is also open-sourcing its effort to optimize 10-bit color performance in the dav1d decoder that it is using to support AV1 on Android.

Rethink Research’s Thomas Flanagan pointed toward Bitmovin’s third annual developer survey, which suggested that one in five developers plan to implement AV1 this year, and that companies including Cisco, Mozilla, and YouTube are scaling up their use of the technology.

But that is not to say there are no roadblocks in the adoption of AV1. However, Amazon – also a founder AOM member – thinks it too early to speculate on AV1’s viability as a competitor to other codec technologies. A good analogy is the evolution of VP9. It is used by new internet companies but not by traditional broadcasters.

Then there is the fact that AVC decoders are embedded nearly everywhere, compared to HEVC encoding in AV1, which was noticeably slower a year ago, as per some benchmark tests. Creating a product with an AV1 decoder will take about 18 months, and it will take longer for support to be ubiquitous. It will happen eventually, but not on the day ratification of the standard is complete.

Other variables that could impact AV1 maturation are: software development can take unexpected twists and turns; and open-source is an open invitation for the development of multiple codecs, each with different configurations and approaches to experimentation.

Hence, just like HEVC is an efficient replacement for AVC, AV1 takes a step forward and promises even higher compression without any significant quality loss. However, the incompatibility issue might be the deal breaker currently. Hence, AV1 may be royalty-free, but it is not indemnification-free against patent claim violations.

Nevertheless, AV1 is well positioned to compete with H.265/HEVC, and to succeed VP9 for open-source use cases in 2020.

Device makers and streaming providers now have a full set of tools for building AV1-compliant products and services.

An alternative to AV1 and HEVC – MPEG-5 EVC
Just as the royalty-free AV1 video codec had started to gain traction for 4K and 8K video streaming, three major players in the industry have announced a plan to promote (MPEG-5 EVC) in the multimedia industry. Apparently not fully content with the new AV1 video codec, Samsung, Huawei, and Qualcomm are backing MPEG-5 EVC, touting it as the next-generation video codec.

By delivering 4K UHD video with greater compression and efficiency over the previous standard codec, MPEG-5 EVC enables more screens to display 4K, 8K, VR, AR, and HDR content, and offer the level of services that consumers have come to expect.

New 8K TVs from Samsung already support the royalty-free AV1, which is required to stream 8K videos from YouTube. Samsung is also a member of AOMedia, which has developed AV1. It is unclear if Samsung’s plans for AV1 will be affected by its commitment to MPEG-5 EVC.

MPEG-5 EVC is designed to be an alternative, not a successor, to HEVC. The MPEG group is developing its next-generation video codec under the name VVC, also referred to as future video coding (FVC) or H.266.

MPEG-5 EVC was released at the end of April 2020. The base layer will be royalty-free whereas the enhanced layer will be subject to patent royalties. All three parties hold patents relating to MPEG-5 EVC, and have committed to offer them on so-called FRAND (fair, reasonable, and non-discriminatory) terms.

For now, AV1 looks like a strong contender, given its support from Amazon, Apple, Facebook, Google, Intel, LG, Microsoft, and Netflix. However, it would not be a surprise if MPEG-5 EVC emerges as an alternative.

Coding-efficient VVC
Versatile video coding (VVC) is a collaboration between MPEG and video-coding experts group (VCEG). It aims for a 30 to 50 percent bit-rate reduction for the same perceptual quality as HEVC, but with an estimated 10 times or more encoding complexity compared to its predecessor.

As its name suggests, VVC is meant to be versatile, supporting applications like gaming and adaptive streaming, a technique that adapts bit rate and video resolution to network conditions. Typically, various applications are supported by different profiles of a standard, and this has been the case for HEVC.

However, in most devices, only the main profile of HEVC has been implemented, leading to a lack of support for wider applications. VVC is expected to tackle a broader range of application scenarios in one profile, which is an advantage over HEVC.

Yet, similar to HEVC, VVC is a royalty-bearing video codec. This means that individuals and companies implementing VVC for their products will need to pay patent licensing fees.

Content-aware encoding
H.264, despite its limitations, continues to constitute the vast majority of video in broadcast and streaming. And it keeps improving through techniques of content-aware encoding (CAE). This is an encoding technique, which uses machine learning to compare content against known parameters for a given device and/or media player type, can boost the picture quality and reduce the distribution cost, and are already proving to prolong the life of MPEG4/H.264 AVC. However, CAE applies largely to file-based transcoding for on-demand content.

CAE is used on existing codecs that rely upon VBR methods in ABR delivery. Harmonic’s EyeQ CAE solution is already commercially deployed for live applications. Netflix introduced this technology 2 years ago. CAE has the potential in AVC to save up to 50 percent bandwidth, but is content dependent and not as powerful as HEVC. Nevertheless, it is a very good replacement.

Content-aware encoding is gaining significant attention, as it can differentiate between different kinds of broadcast content, either static talking-head style interviews or more dynamic sports images, to ensure that the quality, latency, and bitrates of each piece of content are properly optimized.

Codec-agnostic LCEVC
Despite being part of the same MPEG-5 family, there is no connection between EVC and LCEVC. It is a fundamentally different approach to EVC, in which the solution relies on taking an existing codec available in hardware – which could be AVC, HEVC, or in future, VVC – and adds a software layer on the top, responsible for improving the performance of the hardware encoder.

LCEVC is a codec-agnostic enhancement coding standard – that is, an add-on, not an alternative – which enhances any other video codec (including EVC) by simultaneously improving compression efficiency and reducing processing power consumption.

The approach is making it possible for companies to deploy hardware video codecs once, and upgrade them with software, often a far more affordable alternative to ripping and replacing devices in the field. The vendor driving this is V-Nova, on whose technology Perseus Plus, LCEVC is based.

Specifically, due to its lightweight nature, LCEVC can be efficiently implemented in software, and also at the application level. This is thanks to its use of standard graphics hardware, which is accessible and power-efficient at all levels in the software stack.

For example, a service currently streaming a mix of H.264/AVC and H.265/HEVC to a range of devices could roll out LCEVC enhancement across all of its services to improve video quality and reduce operating costs, without changing any equipment or infrastructure.

The business cases involved in LCEVC tests typically include improvement of QoE, expansion of customer reach, reduction of churn, reduction of delivery costs, and reduction of infrastructure costs, all at the same time.

For these reasons, LCEVC has great potential to be adopted much more rapidly than traditional standards that typically require many years for new hardware components to be designed and manufactured before they even begin to be distributed in new devices.

The intention is to publish license costs for LCEVC as soon as possible after final publication of the standard. It is anticipated to be a transparent, accessible, and clear licensing structure to enable quick and wide adoption by the market players.

The codec of choice
When it comes to broadcast and pay-TV operations, video compression is essential in delivering content to audiences. However, as the demand for high-resolution video increases, the need to advance video-compression techniques is becoming unavoidable.

Streaming the entire 360 panorama of VR and AR content smoothly in high quality on a wide range of devices requires a huge amount of bandwidth. High-quality VR video requires 15Mbps and upward of 25Mbps, depending on a number of variables, such as internet speed and content resolution (HD versus UHD).

More often than not, the results are mediocre quality and viewing experiences. Just as important – if not more so – than new codec technologies in this regard are new methods of content production and distribution, such as viewport-adaptive encoding and streaming of 360-degree video for VR and VR tiling technology for transmitting complete UHD fields of view.

Hence, the industry must also develop an entirely new codec for massive data to handle applications from VR and multiple UHD streams to light field. Nonetheless, MPEG, ISO, and ITU are exploring the need to include omni-directional video-coding technologies in a future video-coding standard.

Given each codec has different properties and that every broadcaster or pay-TV operator has different infrastructure and legacy systems, it is unlikely there will be a clear winner in the codec domain. Instead, there will be several dominant players per niche.

Improving the landscape of codecs – both old and new – hinges on the learnings and technological innovations taken from ongoing advancement works. So, whether viewers are watching video content in 8K or HD, every audience will benefit from continual enhancements; future-proofing the viewing experience and ensuring that no consumer group gets left behind.

The winner will be the codec that best solves a problem in a given application. This is going to be measured along the dimensions of compression performance, computational performance, and compatibility.