Building the World's Largest IP Broadcast Facility

In 2016, NBC Universal started building a new global headquarters for Telemundo Enterprises, in Miami, Florida. With 13 production studios and seven control rooms, it was to become the world’s largest IP broadcast facility using the SMPTE ST 2110 environment. The paper in the May edition of the SMPTE journal provides a detailed look at the engineering design challenges. Here is an overview of what this means in terms of infrastructure, the decisions made in the process and the lessons learnt along the way.

IP broadcast

Internet Protocol (IP) broadcast provides a cost effective and reliable way to move audio and visual footage from broadcast studios to control rooms. Unlike SDI connections, IP broadcast is uncompressed to provide the best quality and is better designed to handle live broadcasts as it doesn’t suffer from the same latency issues associated with uncompressing the data for use.

The SMPTE ST 2110 provides a set of standards for the delivery of digital video over an IP network in an uncompressed form within broadcast and production facilities. The audio, video and ancillary data are all carried in separate streams, which simplifies the process of adding captions, subtitles and time codes. There is specific guidance available for all the sections covered by ST 2110, including the delivery and timing, audio and use of ancillary data packets.

Telemundo Center facilities

At its launch in 2018, the Telemundo Center was the largest ST 2110 environment in the world, with over 12,000 unique HD sources and 150,000 multicast streams across audio and video. The 500,000 sq ft space features production facilities for news, sports and scripted entertainment, delivered both for broadcast and digital platforms.

Inside the facility is a total of 13 production studios of varying sizes up to 8,000 sq ft, all feeding into one of five live production control rooms (PCRs). There are a total of 72 edit seats, roughly half of which are edit rooms, and a further 60 seats for graphic creation. There are also central areas for video playback, graphics playback and camera shading. All of this is served by a central equipment room with a core of IP video routers using the SMPTE ST 2110.

Master control and distribution is handled outside of the facility by other NBC Universal properties, and for transport the signals are converted to the compressed SDI format. The same is true for any incoming signals to Telemundo Center – these signals are converted from SDI to IP when they enter the facility.

Technical choices

In creating the infrastructure for the facility, the engineers had to make a series of choices about the technology that was to be used. There are two main classes of architecture that are used for the video network, a leaf-spine or single tier topology.

Leaf-spine offered the advantage of avoiding wasting bandwidth by aggregating the endpoint devices and the ability to simplify the cabling by locating the switches close to the endpoints. However, the single-tier system has a simpler and more cost-effective structure. The facility chose to install two single-tier networks, for production and acquisition, each serviced by a large core switch with over 2,000 10-GigE ports.

The Telemundo network is a software defined network (SDN). This means that it uses a software-controller, rather than hardware to control the core switches to route the video flow.

There are actually two separate SDN networks running to provide redundancy, with seamless switching based on the SMPTE ST 2022-7 environment. In this environment, both networks actively transport the video, using a timestamp and sequence numbers. It is this redundancy that is one of the biggest benefits of IP compared to SDI connections, though it still requires a robust monitoring process to avoid critical system faults.

Timing is everything

Rather than using an analog video signal as a reference to synchronize the video created throughout the facility, Telemundo Center uses a precision time protocol (PTP). The PTP is a two-way communication protocol that delivers highly accurate clock information across the network.

Though not video-specific, there is an SMPTE PTP profile for professional media networks. Each PTP clock can only serve so many endpoints, so for a system as large as this, it uses four grandmaster PTP clocks that generate a master signal from a GPS or other external source. Multiple boundary PTP clocks then act as submasters to connect the entire network.

Futureproofing 

One of the big advantages of an IP network is that it is future-proof for a higher resolution broadcast in the future. While in the early days of digital broadcasts SDI was initially better equipped for HD and 1080P feeds, IP has now surpassed its available speeds.

While currently broadcasting mostly at 1080i 59.94, Telemundo’s system has been designed with 50 percent reserved bandwidth, which would allow it to run 1080p if required. These ports could also be run in groups of four to allow enough bandwidth for a 4K operation.   

One consideration for an IP network is the efficiency of the bandwidth available. While running just one or two streams of 1.5Gb/s (1080i) through a 10GbitE port may be wasteful, an over aggregation of streams through the use of switches would unnecessarily complicate the system. With plenty of ports available for expansion, only a light level of aggregation was used at Telemundo, which enabled the core switch design to remain simple.

Audio treatment

With SMPTE ST 2110, audio and visual feeds are transported separately, so the Telemundo Center had planned to use AES67 networking for all of the audio. However, they found that for such a large-scale system it was better to use an audio-only router for the production audio. This used a bank of bi-directional ST 2110 to MADI converters between the audio production consoles, studio preamps and in-ear monitors.

Virtual sources

As everything in the facility is connected through the IP network, there is no distinguishing between local and core sources to the production control room. It also means that sources don’t need to have globally unique names as they would using SDI.

These virtual sources can be categorized by their destination. Rather than PCR 1 using CAM 1 and CAM 2, then PCR 2 using CAM 3 and CAM 4, each studio will always have a CAM 1 and CAM 2. If viewed from a different control room, the camera name is preceded by the control room number. For instance, CAM 1 going to PCR 1 will be called PCR 1 CAM 1 if viewed from PCR 2.

Though an end-to-end IP network is the ideal solution on paper. In practice, the Telemundo Center chose to use SDI connections for its front-of-camera connections, such as on-set monitors and LED screens. This was down to the flexibility it offered, if the studio needed to be quickly changed. The SDI to IP gateways were placed locally in the studio network closets, to avoid excess cabling.

Conclusions

Being the largest IP network project of its kind, the building of the Telemundo Center has produced some useful lessons for future developments. Though there were specific issues, the biggest takeaway was perhaps that, while a perfect IP system is ultimately the aim, it is more than acceptable to make concessions along the way to create a system that works for you.

However, keeping the system uncomplicated is also key. This includes avoiding legacy SDI coax wiring practices such as jackfields and DAs to reduce unnecessary rack space and passive gear in equipment rooms.

Quality control was also an important lesson, from the cleaning and inspection of the fiber-optic cables prior to installation, to the on-going troubleshooting once in operation. And in the planning process, the configuration of the IP network can be more complex than the installation, due to the complexity and bi-directional nature of the signals, so it’s important to leave extra time for configuring the software and network flows. 

On completion, there were still a few final components that needed to be installed to create a fully operable ST 2110 system. These included the adoption of a common stream connection management protocol across vendors, the adoption of the advanced SMPTE ST 2110-40 ancillary data stream processing and audio stream packaging standardization.

However, the successful completion of this project proved that a complete ST 2110 environment was not only viable but that it offers huge potential for the future. It is highly likely that more broadcast stations will follow suit in the near future.