See How it Works
Learn why technologists and engineers join
Learn why technologists and engineers join
View the companies that support us
Access information about your local section
Access information about your student chapter
Learn more about SMPTE standards
SMPTE engineering documents and others
Get involved in creating next generation standards
Online and in-person meeting schedule
Operations manual and guidelines
From the media industry’s perspective, the new pandemic-altered landscape has “had a very big effect on roadmaps for the video compression industry,” opines veteran industry consultant Ian Trow, SMPTE’s UK Section Manager. In particular, Trow elaborates, “the pandemic has sharpened things up. The social media/video conferencing aspect of everthing has really come to the fore. People have been forced to adopt remote working, video-conferencing solutions. That’s why I say the pandemic has had a big effect on development. It’s accelerated some features that previously were slowly progressing, and certainly put a lot of focus on very specific areas—both for video compression and for how broadband networks handle that kind of traffic generally.”
In particular, Trow believes the acceleration of social-media and video conferencing communication as a result of the pandemic has created an opportunity for optimizing and improving various compression schemes, and creating new business opportunities in that space.
“At first, social media and video conferencing pretty much just took the compression they were given to begin with, particularly [the widely established codec] H.264 [AVC],” he says. “But now, people are starting to realize there is a lot of optimization that can be done, and a lot of tuning of existing compression codecs and future codecs to specific applications to basically minimize the bandwidth that is required, while optimizing the performance.
“Right now, if you think about it, you get a reasonable experience in terms of video and audio through video conferencing, but if you look at some of the applications, it can be quite shocking when you put up more than five or six sources, let alone the 50 sources that represent the upper limit for participants. With the need for audio/video synchronization, the sustaining of services, the fact that video is essentially a unicast media, not broadcast, in this application—all those things mean that network provision and network capacity can be a pretty scarce thing. Therefore, a lot of big social media companies are starting to realize that if they can gain a performance edge in those areas, it can make their platform look that much better. In fact, that is one of the assignments I often get [as a consultant] these days—looking at existing standards and toolsets, and working to specifically optimize them for video conferencing.”
Trow emphasizes that, for these kinds of applications, H.264 “very much remains the incumbent, and is the bedrock of HD streaming and broadcast.” However, as he discussed in a technical paper he presented at IBC 2018 and republished in the SMPTE Journal earlier this year, “the new kid on the block” is the open-source, royalty-free format known as AV1. As compression options become refined, and others enter the fray, he emphasizes that the key issue to keep in mind is not so much which codec is “better” or “more practical” or “more popular,” but rather, that there has been what he calls “a change of emphasis” in terms of what makes such compression schemes valuable to the industry under current and foreseeable circumstances.
“The old incumbent H.264 is ubiquitous right now, but the key is that the environment has changed,” he explains. “Whereas, it used to be about set-top boxes and chips, it’s now about browser compatibility and support on devices like laptops and smart phones. Thus, while the conversation used to be about hardware-developed encoders, it is now about software encoders, looking for run-time and processor efficiency, because those are the types of things that are now important in this new world as you widen the area of application for many video compression standards.”
Late last year, Trow participated in a SMPTE Webinar on the topic of various applications for new compression standards. He says when you dig deep down, there are as many as 500 different parameters to consider in making media applications truly interoperable with a wide range of browsers, player applications, and operating systems for streaming applications.
“This explains why there is so much inertia behind Codecs that have already been widely deployed—namely gaining compliance and interoperability across a vast area of operating systems, devices, and applications,” he says. “Interoperability was already a significant issue within the classic linear broadcast scenario. With the explosion of social media applications, playout platforms and devices, achieving adequate coverage among so many more variables are understandably an overwhelming prospect where compromises have to be made. Consequently, interoperability is often achieved incrementally through a trial-and-error process, rather than a pre-determined scheduled interoperability phase. No surprise then that upgrading an operating system, browser, or app, let alone changing a device, is often the first time serious interoperability issues come to light.”
Additionally, he points out that in today’s world, compression standards need to be viewed as “data driven,” rather than thought about as “video” standards. This is true for all applications, he says, but is particularly important for social media and video-conferencing applications, simply because of the enormous impact video can have on those watching or interacting with it these days.
“That’s another thing that is subtly different from the way such standards were developed in the past,” he says. “They are now about the issue of tagging, metadata, clip discovery, and so on. People need to have confidence that they are seeing authentic footage, and a metadata audit trail, along with watermarking, are techniques used to do that. I think a lot of people still think video is primary, audio is secondary, and data is a distant last. You can forget that now. In this new world, the actual metadata that accompanies particular content to make it searchable is of prime importance. Services are only as good as how searchable their content is, what kind of metadata they offer, and how it can actually streamline the subsequent processes afterward.
“So I think one of the key things for any compression standard being developed is how well they operate within a data-driven application. To widen the appeal, video compression certainly relies upon being able to handle that data and the sort of authentication type aspects that people demand to be able to search the content and believe that it is authentic.”
Therefore, Trow points out that much of today’s work on compression standards revolves not so much around making new and more complex standards as making existing standards work more efficiently on what he calls “low-complexity platforms”—social media style applications, in other words. This is why so much effort in recent years has been put into low-complexity encoding, such as the MPEG-5 LCEVC [Low Complexity EVC] standard. Trow expects that “interesting development” to continue.
It’s also an important development, he continues, because radically rising congestion across private, personal, public, and business networks these days is, according to Trow, “sometimes causing streaming companies to drop their quality.” He points to announcements earlier this year as the pandemic was roiling the globe by Netflix, YouTube, and others that they would reduce streaming quality in certain European markets to reduce strain on broadband networks.
“That was being done to basically alleviate the impact of their content on networks,” he explains. “So on the one hand, we are seeing an explosion in the rate of content being consumed, but on the other hand, most of that content is not live right now—it’s catchup, video-on-demand, or player-based content. The kind of applications that people are using are favoring file or packaged delivery. But concern is being expressed about what happens when live-event and sports broadcasters start cranking up as COVID restrictions relax. Content consumption is likely to increase not only from streaming services from broadcast incumbents or content aggregators, but the redistribution of content via social media. That is really driving current development. There could be damage to brands if there is poor technical performance, and there is usually a churn rate associated with a poor startup experience, for example. A lot of operators are now becoming acutely aware of this. They think it is great that people are really interested in their services, but they feel they have to step up in terms of the technical performance not only of their compression approach, but the overall network architecture.”
Which leads to the ongoing debate over quality versus immediacy. Trow points out that poor video quality in streaming content in the past was often forgiven by consumers because of the immediacy of a live event or breaking news or social experience. “But now, we are talking about using video remotely, including remote video production,” he elaborates. “So people have higher expectations for higher levels of service and are not going to be so forgiving. That creates a competitive edge between various providers. Because of that, there is sort of a pecking order in terms of the quality of a video codec’s performance and the subsequent services that are now being used within a broadcast production environment.”
This is among the reasons why the industry has become extremely interested in seeing how it can weave machine learning/AI techniques into the compression process, according to Trow.
“In a [simple social-media based video conversation], as much as 90 percent of the region around [the person speaking] is not changing,” he relates. “Large parts of the picture are actually static. So there is a lot to be gained from using AI and machine learning to actually look at how the codec is operating to guide toolsets to recognize areas of activity within a picture. Similarly, there is a lot of potential advantage in driving encoding engines to examine what is actually a synthetic scene. A lot of codec parameters today are governed by the [presumption] it is a natural scene. But for social media and video conferencing, the characterization of the type of scene that you are looking at, plus the degree of complexity, is often very different from typical broadcast material.”
This is why Trow expects we will eventually see what he calls “incarnations of the current standards where the toolset that is actually used within the standard is selected and weighted according to the specific application. Up until now, most people have accepted vanilla versions of the code for encoding. But now, when media traffic has such a significant impact on network loading, anything that can be done to alleviate network congestion and ease latency aspects of delivery is being eagerly explored.”
In other words, as Trow puts it, “not only is the optimization of video compression in itself a big issue in driving new standards, but actually so is the handling of [the compression codec] based on the application.”
Indeed, in a sense, “the application” seems to be everything these days. In the crucial, rapidly growing area of remote production, for example, the new lightweight, low-latency algorithm JPEG-XS has come along to offer near lossless quality to producers of live in-the-field projects, such as sporting events. Other options are out or on the way, as well, as Newswatch discussed late last year.
Of course, these and other developments could be slowed down by the industry’s current big roadblock on the business side—the debate over licensing and royalties that has hindered the proliferation of HEVC/H.265 [High Efficiency Video Coding] as a successor to AVC. Trow points to “uncertainty” across the industry regarding how to handle royalties for such standards.
“Naturally, a lot of contributing companies want to monetize their investment in development of such standards,” he says. “The industry seems to have reached an impasse where, previously, there was always a standardized encoder—typically an MPEG encoder—and a proprietary encoder. Market forces would determine what royalty and patents were payable to what compression engine based on market rates. For a long time, that seemed to help the industry. But what seems to have happened now is that has broken down with things like HEVC. In a classic environment in the past, when H.264 was so successful, there were periodic licensing and royalty wrangles, but they pushed through and came up with a business model and licensing strategy that was successful and allowed the codec to establish itself. Then, HEVC came along and people had different expectations out of it and the landscape was not the same.”
He explains that HEVC was developed in a world with ultra high-definition at the top end of the broadcast performance spectrum and online video streaming on the other end. “That means you would expect to get an increase in resolution and quality, but also features to support high dynamic range and so forth,” Trow continues. “So a lot of those features were added to HEVC. But on the [low end], that wasn’t needed, and that caused problems in the marketplace in terms of what people wanted to pay for royalties and licensing. In the streaming domain, they didn’t want to adopt something with a complex and unwieldy licensing and royalty scheme.”
“That is one reason that a lot of people who are focusing on Internet and social-media type applications formed the Alliance for Open Media [AOM] and went ahead and developed license-free AV1, which is more along the lines of what they expect and need for enterprise type applications. The aim of AV1 is to achieve the kind of performance that the Internet-based technology companies wish to see out of video streaming. So there is a definite divergence in terms of how compression is currently being developed, and for what applications.”
Other streaming options like MPEG-DASH and Apple’s HLS also exist, and undoubtedly, more are on the way. But the point, as Trow mentioned, is that the so-called “high” and “low” platforms for displaying video seem to have diverged in terms of what their compression needs are. But, in either case, no matter what might come next, it’s clear that innovating with video compression remains the foundation of the larger video display ecosystem, though perhaps in a more specific way than in the past.
“In the pure algorithmic development area, the development is being done for specific applications,” Trow says. “It’s being funded by parties that are interested in seeing better streaming, better social media and video-conferencing type applications. Now, people are trying to dig deeper than in the past in this regard, like trying, for example, to come up with an approach that particularly maps itself for service-based delivery in the Cloud. So I believe the compression industry can be very successful if it understands the application it is targeting. For the video world, a big part of it is understanding how to make video ecosystems and infrastructures that are based on compression, but take into consideration a whole host of parameters that are required for successful service. A case in point being medical applications for [video transmission]. Image quality, low latency, outright video performance and fidelity are being used to help people make actual life-and-death decisions and diagnosis these days. This represents a significant shift to achieve a beneficial use of this technology that is way beyond the initial media distribution application. It not only relies upon outright video fidelity, but a comprehensive understanding of the application and the associated metadata.”
Earlier this month, SMPTE announced the names of the honorees it will celebrate at an interactive version of the organization’s awards gala as part of this year’s SMPTE 2020 remote technical conference. The honorees will be recognized on 11 & 12 November after new SMPTE Fellow elevations are conferred on 10 November. Among those slated to be honored are legendary visual effects’ supervisor Richard Edlund, ASC, with Honorary Membership in SMPTE in honor for his work advancing the art and science of visual effects’ cinematography; the late Natalie Kalmus, who will receive the Progress Medal posthumously in recognition of her pioneering contributions to motion-picture color, and who will have the Technicolor-Natalie and Herbert T. Kalmus Medal renamed in her honor, with that award going this year to Beverly Joanna Wood for her contributions to the development of the Color Contrast Enhancement (CCE) and Adjustable Contrast Enhancement (ACE) processes; and Rich Chernock, who will receive the David Sarnoff Medal for his contributions to the development of the ATSC 3.0 digital television standard. See the full list of honorees here.
A recent story in The Hollywood Reporter suggests major studio remote collaboration initiatives are accelerating in the time of COVID-19. The report discusses the formation of a strategic partnership between Universal Filmed Entertainment and Microsoft for the purpose of using Cloud-based technologies “to build new live-action and animation production processes.” The story says that Universal will expand the private remote collaboration network its subsidiary, DreamWorks Animation, now uses onto the Microsoft Azure cloud platform for the purpose of widening the studio’s ability to utilize a larger global base of talent, and then expand that same system for use on major Universal tentpole productions. The initiative was planned long before the global pandemic, but the studio has particularly prioritized Cloud-related workflow technologies in recent months. Additionally, Universal CTO and Senior VP Michael Wise is quoted in the article as saying one of Universal’s goals is to develop non-proprietary approaches in hopes of developing “a solution that works broadly for the industry.”