Popular Science: What is 400G Ethernet?

With 5G, the rapid development of the Internet of Things (IoT), the explosion of social media traffic, the shift from traditional to remote work… All these changes are accompanied by the pressure of rapidly growing data and bandwidth-intensive applications, data The center faces a capacity upgrade.

In addition, data shows that in 2021, about 94% of workloads and computing will be processed by cloud data centers, while traditional data centers only handle 6%. Bandwidth requirements for cloud data centers are doubling every 12 to 15 months, even outpacing the growth rate of the wide-area Internet. All of these drivers are driving growing data center traffic and the need for high-capacity data center networking solutions.

Currently, 100G is the fastest widely adopted Ethernet connection and is growing steadily, however, with the continued growth of traffic and the demand for high-capacity data transmission such as video, 400G Ethernet will become an inevitable trend in data centers .

What is 400G Ethernet?

The past decades have seen a shift in Ethernet speeds from 1G, 10/25G to 40/100G, and the Ethernet industry continues to innovate to enable higher network speeds such as 400G and 800G.

400G Ethernet, or 400 Gigabit Ethernet (400GbE), was developed by the IEEE P802.3bs Task Force in 2017 and uses roughly similar technology to 100 Gigabit Ethernet. 400GbE technology requires the ability to process more symbols per second and advanced modulation formats to deliver 400Gb/s per wavelength transmission capacity at the best cost-per-bit, with smaller form factors, fewer points of failure and interfaces, and more. Low power and heat – all at 4 times the capacity of 100Gb/s.

Challenges and Trends of 400G Technology Development

The challenges of 400G technology development include the following four aspects:

Challenge 1: DCI connectivity. Data shows that from 2019 to 2023, global interconnect bandwidth is expected to grow at a compound annual growth rate of 45%. Factors such as the growing popularity of CDNs, the rapid development of cloud services and the need to transfer data between clouds, and the increase in the amount of data that needs to be replicated across data centers, have contributed to the high growth of DCI traffic.

Challenge 2: Uncompressed 4K/8K/16K video. The COVID-19 pandemic has accelerated the development of video as a means of communication, with more than 80% of internet traffic now being video. This means that the network must provide higher bandwidth to transmit uncompressed video.

Challenge 3: HPC clusters. High-performance computing (HPC) clusters are proliferating in data center environments and pose significant scaling challenges. HPC clusters will undoubtedly increase the demand for 400G data center switching capabilities.

Challenge 4: Storage and memory. In terms of storage and memory capabilities, 400G data networking is critical to simplifying the adoption of non-volatile memory express (NVMe) capabilities in the data center, while memory-optimized latency may hinder 400G deployments.

The following technological developments address these challenges:

Trend 1: QSFP-DD. QSFP-DD technology is addressing the need for improved high-density networking solutions in a rapidly growing industry. QSFP-DD represents the first time ever that all ranges, from copper cables to long-reach optics, can be offered in the same common form factor. The QSFP-DD Multi-Source Agreement (MSA) team also released revision 4.0 of the Common Management Interface Specification (CMIS) for the QSFP-DD form factor.

Version 4.0 of CMIS was developed to enable the use of a common code base across products and modules, and also addresses the shortcomings of previous versions of the pluggable optical management system. CMIS also helps ensure that QSFP-DD modules are backward compatible with all QSFP-based transceivers, including widely deployed 40G/100G/200G products.

As 400G adoption grows, QSFP-DD provides the most compact 400G module with the highest port bandwidth density for the data center market. Through rigorous testing across the industry, the QSFP-DD function has been validated as a high-density, high-speed pluggable module best suited for 400G data center network environments.

Trend 2: Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE). 400G data networks are the key to accelerating the adoption of RoCE technology in data centers. RDMA supports zero-copy networking by enabling network adapters to transfer data directly to and from application memory without copying data between application memory and the operating system’s data buffers. The RoCEv2 protocol also supports RDMA across L3 boundaries.

Due to the RoCE structure, data center network switches provide wire-speed performance by enabling Explicit Congestion Notification (ECN) directly, resulting in a performance boost. Additionally, RCoE helps ensure low latency, intelligent congestion management and QoS flexibility.

Trend 3: Smart NIC (SmartNIC). The advent of SmartNICs played a key role in enabling 400G to meet the new massive bandwidth demands of the data center. The latest smart NICs offer two 25G/50G/100G ports or one 200G/400G port and Ethernet connectivity with 50G PAM4 SerDes capability and PCEi 4.0 host connectivity. The availability of SmartNICs is integral to increasing network programmability and accelerating application offloading, which is critical for improving data center efficiency.

Trend 4: DCI connections with 400G ZR/ZR+. 400G ZR is a standard designed to transport multiple 400G payloads over DCI links using DWDM (Dense Wavelength Division Multiplexing) and higher order modulation. The 400G ZR+ enables deployments over 80 km and is ideal for metro area applications such as 5G backhaul traffic, 5G Fixed Wireless Access (FWA).

Essentially, 400G ZR was developed to ensure high bandwidth and high density DCI at the same time. Approved by the Optical Interconnect Forum (OIF), 400G ZR is critical to advancing interoperability and reducing the cost and complexity of pre-400G DCI. The 400G ZR in the QSFP-DD form factor is expected to support increased adoption of routed optical networks in metro applications, including DCI and Distributed Access Architecture (DAA).

Which verticals are driving the development of 400G?

The following key verticals play an important role in driving 400G demand:

1. Communication Service Provider (CSP). Communication service providers are leading the way in driving 400G adoption. Strategically, CSPs are committed to deploying 5G for both consumer and enterprise customers, increasing bandwidth requirements across the network. It also has to have the density of RAN/small cell deployments to meet its 5G business goals. Communication service providers are accelerating the adoption of 100G/400G architectures, and 5G builds require moving cloud resources closer to users and applications and devices. As a result, edge computing resources must support many more devices across the network, as many as 100,000 active devices per square kilometer. In contrast, 4G/LTE can only support up to 2,000 active devices in a square kilometer. With 400G bandwidth, 5G is able to expand the devices and applications needed for its mainstream adoption.

2. Hyperscale cloud providers. The dramatic increase in bandwidth demands from cloud providers’ data centers is a key driver of 400G adoption. Hyperscalers are transitioning to 50/100G NICs, expanding network programmability and enabling application offloading critical to improving overall data center efficiency, including reducing power consumption. With data centers accounting for 1% to 3% of global electricity consumption, hyperscalers are investing heavily in improving Power Usage Effectiveness.

3. Large enterprises. Large enterprises are accelerating the adoption of 400G technology as part of their network strategies for the next three years or more. The main driver of this strategy is the widespread adoption of AI/ML computing clusters that require 50/100G I/O and 400G scaling. These capabilities are critical to businesses.

4. Professional media network. Uncompressed 4K/8K/16K video requires more bandwidth and less latency to meet video professional standards. Since 2020, video traffic has surged, and video streaming services have grown by more than 70% from their peak. The explosive growth in video application capacity demands and bandwidth outstrips current transmission capabilities, and 400G has become critical in meeting the expansion and QoE demands of the video networking space.

400G manufacturers

Cisco

Cisco Nexus 9000 Series product support provides a comprehensive 400G solution. Cisco has announced that its Nexus family of related 400G products will meet the needs of all network architectures, will cost-effectively provide high-density 100G/400G support for customers with large network scales, and the entire system will have higher flexibility.

The Nexus 9300 and 9500 are the next-generation, non-modular products of the Nexus 9000 series that support 400G. Based on Cisco Cloud Scale technology, the series introduces fully backward compatible 400G optical interface-Quad Small Form-Factor Plug-Double Density (QSFP-DD) ports that can transparently scale existing data center fabrics from 40Gbps and 100Gbps Speeds migrated to 400Gbps. With highly flexible Layer 2 and Layer 3 scalability, and high performance to meet the changing needs of virtualized data centers and automated cloud environments.

Nvidia

In April 2022, NVIDIA released the next-generation Ethernet platform NVIDIA Spectrum-4. Comprising the NVIDIA Spectrum-4 switch family, ConnectX-7 SmartNICs, NVIDIA BlueField-3 DPUs, and DOCA data center infrastructure software, the platform can dramatically accelerate large-scale cloud-native applications.

Spectrum-4 claims to be the world’s first 400Gbps end-to-end network platform. Its single-chip switching throughput reaches 51.2Tbps, which is 4 times higher than the previous generation. It can provide ultra-high network performance and Strong security. The SN5000 switch powered by Spectrum-4 can support up to 128 400GbE ports or 64 800GbE ports. Spectrum-4 also optimizes the RoCE network architecture and provides Adaptive Routing and enhanced congestion control. The encryption bandwidth reaches 12.8Tbps and can support hardware-level MACsec and VXLANsec.

New H3C

In April 2022, H3C released the 400G campus core switch H3C S10500X-G. The S10500X-G relies on data center-level network technology and a new design concept to break the performance limit of campus switches. The S10500X-G adopts Comware, a cloud-native network operating system independently developed by H3C. It has innate cloud-based, container-based, and micro-service-based core capabilities. And the application of AI ECN technology achieves high data throughput and low latency. In terms of security, the S10500X-G switch supports the full-port hardware-level encryption technology MACSec (Media Access Control Security, MAC security), and the key length can reach 256 bits, which protects the Layer 2 protocol from attacks from the root, and reduces data leakage and suffering. Risk of malicious cyber attacks.

The 400G era is coming, are you ready?

There is no doubt that the development of 400G Ethernet has become inevitable.

For manufacturers, opportunities and challenges coexist. Today, major suppliers have launched 400G products one after another, and it is not easy to win from the fierce 400G market competition. However, for users, with the standardization, scale and mass production of 400G products, the cost of 400G construction will gradually decrease, and users will have more choices. In addition, with the sharp drop in the deployment cost of 25G and 100G, it is undoubtedly beneficial for users who have an urgent need for 400G or users who are on the sidelines.