Content-aware encoding — why every little bit helps

SophiaFiorino
Monday, July 27, 2020 15:28

Image of YouTube on a smartphone

We're watching more digital video than ever before — an average of 84 minutes a day, to be exact. Whether it's "Tiger King" on Netflix or cat videos on YouTube, our viewing habits have changed dramatically over the last few years, with traditional broadcast media on the steady decline. However, digital video is not without its issues...

  • Buffering. (The all-time worst.)
  • Videos that don't (or won't) load.
  • Videos that look like someone recorded them with a potato. 

How a video looks and performs often influences our enjoyment of the actual content. This is something called quality of experience (QoE). Content-aware encoding — or CAE, for short — is one of the most revolutionary video technologies ever because it significantly improves QoE. 

Encoding Basics

Ever wondered why some online videos go all blurry? It's probably down to the encoding. This is the process of converting and compressing digital files from one format to another.

Videos need to meet the correct specifications and formats for the device they are playing on; otherwise, they won't play properly. Engineers encode videos for different platforms and different contexts. 

Engineers have to create multiple versions of the same video with different bit rates for proper playback in various contexts. This encoding takes time and costs more to store all the versions. Imagine there was a simpler, cheaper way to do all of the above...

Introducing content-aware encoding

Content-aware encoding — sometimes called context-aware encoding or content-adaptive encoding — optimizes different types of digital video content. It achieves the optimum quality and latency for each video type using the fewest bits necessary. It does this via machine learning and deep video analysis. 

Essentially, CAE revolutionizes the digital content we love to watch on computers, TV, mobile, and in the cinema — bit by bit by bit!

How did we get here?

Digital video provides us with an alternative to the traditional TV channels that have dominated our collective viewing habits for decades. Over-the-top services (OTT) — think Netflix, Disney+, Hulu, etc. — have become part of popular culture, spearheading a watch-what-you-want-when-you-want broadcast model that offers more flexibility and choice than over-the-air TV networks. 

Streaming might be at an all-time high, but many OTT providers struggle to make a profit. Cinema broadcast rights and increased competition are among the reasons, but the cost of encoding is another important factor. To keep up with the latest video formats — 4K, HDR, 8K, HFR! — requires a massive undertaking in encoding. And there’s bandwidth, storage and delivery costs to consider.

For years, good video compression software has reduced costs and improved consumers' QoE. But for increased market share, OTT providers need to offer countless formats and unparalleled video quality. Traditional video compression software can't keep up with this demand. 

CAE is so effective because it uses a different approach to encoding. Until recently, engineers encoded videos using 'bitrate ladders,' which present the following problems:

  • When a video has too low a bitrate, the video appears fuzzy to viewers.
  • When a video has too high a bitrate, the video buffers continuously. 

CAE does something different. It analyzes a video's characteristics and optimizes encoding based on these characteristics. It's a much simpler way of doing things.  

How does CAE reduce costs?

Research shows that CAE reduces many of the costs associated with video playback. Primarily, it decreases the number of 'bits' in a file without significantly reducing quality, to maintain a viewable quality even with a poor connection. This allows OTT providers to have fewer bandwidth and storage costs.

CAE examines content during or after encoding, so viewers receive the optimum-quality version of a video at the lowest (practical) bit rate. It does this via machine learning and video analytics, where clever algorithms determine a video file's characteristics and preserve quality.

Many factors influence playback costs — the viewer's location, connection and device type. But, in almost every circumstance, CAE generates high-quality video without load-up and buffering issues. 

CAE also differentiates between different content types. These include sports games — which require higher rates to cope with movement — to bit relatively static interviews. CAE converts all kinds of TV and cinema files into the correct format. It's no wonder, then, that so many OTT providers are now using this technology. 

The future of CAE

Netflix integrated CAE into its platform several years ago, prompting other OTT providers to follow suit. 

"Imagine having very involved action scenes that need more bits to encapsulate the information versus unchanging landscape scenes or animation that need less," said Netflix at the time. "This allows us to deliver the same or better experience while using less bandwidth, which will be particularly important in lower bandwidth countries." 

YouTube also uses (a type of) to CAE to optimize the viewer's experience:

"We care about the quality of the pixels we deliver to our users. With many millions of devices uploading to our servers every day, the content variability is so huge that delivering an acceptable audio and video quality in all playbacks is a considerable challenge."

Looking forward, it's clear that more OTT providers will use CAE to automate and streamline video playback and reduce many of the problems associated with compression and conversion. 

Thanks for reading! SMPTE is a professional membership association of entertainment technology experts. Become a member here, or learn more here!