Shulou(Shulou.com)11/24 Report--

Hello, everyone. I'm Mr. Jujube.

In today's article, let's talk about two of the latest cool techs items in the data center-NPO / CPO.

The story still has to start from the beginning.

Last year, the state issued the strategy of "counting the East and the West", which attracted the attention of the whole society.

The so-called "counting the east and counting the west" is actually the adjustment of the task division of the data center. We have transferred some of the computing needs of the eastern coastal areas to the data centers in the western region.

The reason for this is that the western region is rich in energy resources and low natural temperatures, which can greatly reduce electricity bills and carbon emissions.

As we all know, the data center is the carrier of computing power. At this stage, we can not do without the digital transformation and digital economy, but also can not do without the data center. However, the problem of power consumption in the data center cannot be ignored.

According to the data, the total electricity consumption of the national data center in 2021 is 216.6 billion kilowatt-hours, accounting for 2.6 percent of the country's total electricity consumption, equivalent to the annual power generation of the two three Gorges hydropower stations and the total electricity consumption of 1.8 Beijing areas.

Such terrible power consumption has put a lot of pressure on us to achieve the goal of "double carbon".

As a result, the industry began to step up research on how to reduce the energy consumption of data centers.

Data center (IDC) as we all know, there is an important parameter index in the data center, that is PUE (Power Usage Effectiveness).

PUE = total data center energy consumption / IT equipment energy consumption. The total energy consumption of the data center includes the energy consumption of IT equipment and other systems such as refrigeration and power distribution.

We can see that in addition to the electricity used in the main equipment, there is also a large part of the energy consumption, which is used in heat dissipation and lighting.

Therefore, the idea of tampering with energy conservation and emission reduction in data centers lies in two points:

1. Reduce the power consumption of the main device

2. Reduce the power consumption of heat dissipation and lighting (mainly heat dissipation)

The power consumption challenge of the █ master device when it comes to the master device, people immediately think of the server. Yes, that's right, the server is the most important equipment in the data center, it carries a variety of business services, there are CPU, memory and other hardware, can output computing power.

But in fact, the main equipment also includes a kind of important equipment, that is, network equipment, that is, switches, routers, firewalls and so on.

At present, the accelerated landing of AI / ML (artificial Intelligence / Machine Learning), coupled with the rapid development of the Internet of things, makes the business pressure of the data center more and more intense.

This pressure is reflected not only in the demand for computing power, but also in the network traffic. The network access bandwidth standard of the data center has increased from 10G and 40G in the past to 100G, 200G or even 400G now.

In order to meet the demand of traffic growth, network equipment itself needs to be upgraded iteratively. As a result, more powerful switching chips, as well as higher-speed optical modules, are all in use.

Let's take a look at the switch chip first.

The switching chip is the heart of the network equipment, and its processing capacity directly determines the capacity of the equipment. Over the years, the power consumption of switching chips has increased, as shown in the following figure:

It is worth mentioning that although the overall power consumption of network equipment continues to increase, the power consumption of a single Bit (bit) continues to decrease. In other words, energy efficiency is getting higher and higher.

And look at the light module.

Optical module plays an important role in the field of optical communication, which directly determines the bandwidth of network communication.

As early as 2007, the power of a 10-gigabit (10Gbps) optical module was only about 1W.

With 40G, 100G, 400G, 800G and even later 1.6T optical modules, the power consumption increases like riding on a rocket, soaring all the way to 30W. You should know that a switch can have more than one optical module, and if it is fully loaded, there are often dozens of optical modules (if 48, that is 48 × 30mm 1440W).

Generally speaking, the power consumption of optical modules accounts for more than 40% of the power consumption of the whole machine. This means that the power consumption of the whole machine is very likely to exceed 3000 W.

One data center, and more than one switch. The power consumption behind this is scary when you think about it.

In addition to switching chips and optical modules, there is another "big power consumer" of network equipment that people may not be familiar with, that is, SerDes.

SerDes is short for SERializer (serializer) / DESerializer (de-serializer). In the network equipment, it is an important device, mainly responsible for connecting the optical module and the network switching chip.

To put it simply, it is to convert the parallel data from the switching chip into serial data for transmission. Then, at the receiving end, the serial data is converted into parallel data.

As mentioned earlier, the capability of network switching chips is constantly improving. Therefore, the rate of SerDes must also be increased to meet the requirements of data transmission.

The increase in the rate of SerDes naturally leads to an increase in power consumption.

In the 102.4Tbps era, the SerDes rate needs to reach 224G, and the power consumption of the chip SerDes (ASIC SerDes) is expected to reach 300W.

It should be noted that the rate and transmission distance of SerDes will be affected by the process of PCB materials, and can not be increased indefinitely. In other words, when the SerDes rate increases and the power consumption increases, the PCB copper foil does not have enough capacity to allow the signal to spread further. Only by shortening the transmission distance can the transmission effect be guaranteed.

It's a bit like a shot put game, where the heavier the shot (the higher the SerDes rate), the shorter the distance you can throw.

Specifically, when the SerDes rate reaches 224G, it can only support a transmission distance of 5'6 inches at most.

This means that under the premise of no technological breakthrough in SerDes, the distance between the network switching chip and the optical module must be shortened.

To sum up, switching chip, optical module and SerDes are the three "power consumption" mountains of network equipment.

According to the data of equipment manufacturers, the network switching bandwidth of the data center has increased by 80 times in the past 12 years, and the underlying price is: the power consumption of switching chips has increased about 8 times, the power consumption of optical modules has increased 26 times, and the power consumption of switching chips SerDes has increased 25 times.

Source: 2020 Cisco and / or its affiliates.All rights reserved.Cisco Public in this case, the proportion of power consumption of network devices in the data center is rising.

Energy consumption of network equipment (red) as a percentage of data source: Facebook-OIF CPO Webinar 2020 █ cooling power consumption challenge the previous Xiao Zaojun described in detail the power consumption challenge of network equipment. Next, let's take a look at heat dissipation.

In fact, compared with the increase in power consumption of network equipment, the power consumption of heat dissipation is the real big head.

According to data statistics, switching equipment accounts for only about 4% of the total energy consumption of a typical data center, which is less than 1 / 10 of the server.

But what about heat dissipation? According to CCID statistics, about 43 per cent of China's data center energy consumption in 2019 was used for heat dissipation of IT devices, which is basically the same as 45 per cent of IT devices themselves.

Even now the country has strict requirements for PUE, heat dissipation accounts for nearly 40% according to the tertiary energy efficiency (PUE=1.5, the limit value of the data center).

The traditional way of heat dissipation (air cooling / air conditioning refrigeration) can no longer meet the business development needs of the current high-density data center. Therefore, we introduced the liquid cooling technology.

Liquid cooling is a new technology that uses liquid as refrigerant to dissipate heat for heating parts. The introduction of liquid cooling can reduce the heat dissipation energy consumption of the data center by nearly 90%. Overall data center energy consumption can be reduced by nearly 36%.

This energy-saving effect can be said to be very powerful, saving 1/3 of electricity directly.

In addition to stronger heat dissipation and less power consumption, liquid cooling also has significant advantages in terms of noise, location (not affected by the environment and climate), construction costs (allowing cabinets to use high-density layout, reducing the footprint of computer rooms) and so on.

Therefore, liquid cooling is now used in almost all data centers. In some liquid-cooled data centers, PUE can even be dried to around 1.1, close to the limit of 1.

Liquid cooling, does it immerse the whole equipment in liquid?

Not necessarily.

Liquid cooling schemes generally include two types, namely, immersion type and cold plate type.

The immersion type, also known as the direct type, is to immerse all the components with high heat in the main equipment into the coolant to dissipate heat.

Cold plate type, also known as indirect type, is to attach the main heat dissipation components to a metal plate, and then there is a refrigerant liquid flow in the metal plate, taking the heat away. Nowadays, a lot of DIY assembles computers, which are cold plates.

The server is liquid-cooled, which is a very mature technology. Well, since liquid cooling is needed, of course, it would be better for the server and network equipment to go together, otherwise two sets of systems would have to be developed.

The question is, can our network equipment be liquid-cooled?

█ NPO / CPO, make a brilliant debut! After laying the groundwork for so much, our protagonist is finally going to make a brilliant debut.

In order to reduce the working power consumption and cooling power consumption of network equipment as much as possible, under the leadership of OIF (Optical Internet Forum), many manufacturers in the industry jointly launched-NPO / CPO technology.

In November 2021, Ruijie Networks, a domestic equipment manufacturer, released the world's first 25.6T NPO cold plate liquid-cooled switch. In March 2022, they released the 51.2T NPO cold plate liquid-cooled switch (concept machine).

NPO cold plate liquid-cooled switch

NPO, English full name Near packaged optics, near packaging optics. CPO, English full name Co-packaged optics, co-encapsulated optics.

To put it simply, NPO / CPO is a technology that "encapsulates" network switching chips and optical engines (optical modules).

Our traditional connection is called Pluggable (pluggable). The optical engine is a pluggable optical module. After the optical fiber comes over, it is plugged into the optical module, and then sent to the network switching chip (AISC) through the SerDes channel.

As for CPO, the switching chip and the optical engine are assembled on the same Socketed (slot) to form a co-package of the chip and module.

NPO separates the optical engine from the switching chip and assembles them on the same PCB substrate.

You should be able to see that CPO is the ultimate form and NPO is the transitional stage. NPO is easier to implement and more open.

The reason for the integration ("encapsulation") is to shorten the distance between the switching chip and the optical engine (controlled in 5~7cm), so that high-speed electrical signals can be transmitted between them with high quality and meet the bit error rate (BER) requirements of the system.

After shortening the distance and ensuring the high-quality transmission of high-speed signals, we can also achieve higher-density high-speed ports and improve the bandwidth density of the whole machine.

In addition, the integration makes the components more concentrated, which is also conducive to the introduction of cold plate liquid cooling.

Inside the NPO switch (after opening the cold board), you can see that the distance between the switch chip and the optical engine has been greatly shortened. Behind the NPO / CPO technology is actually the very popular silicon technology.

Silicon light is a silicon-based optoelectronic large-scale integration technology with photons and electrons as information carriers. To put it simply, a variety of optical devices are integrated on a silicon-based substrate to become an integrated "optical" path. It is a miniature optical system.

The fundamental reason why silicon light is so popular is that microelectronic technology is gradually approaching the performance limit, and the traditional "electric chip" is becoming more and more inadequate in terms of bandwidth, power consumption and delay, so it has changed to the new track of "(silicon) optical chip".

The development of █ NPO / CPO switches NPO / CPO technology is a hot research direction of major manufacturers at present. Especially NPO, because it has the best open ecology, the industrial chain is more mature, and can obtain the fastest benefits of cost and power consumption, so its development is faster.

The 25.6T silicon optical NPO cold plate liquid-cooled switch of Ruijie network was mentioned earlier.

This NPO switch is based on a 25.6T switch chip of 112G SerDes, the height of 1RU, and a 400G optical interface with 64 connectors on the front panel. It consists of 16 1.6T (4 × 400G DR4) NPO modules and supports 8 ELS / RLS (external laser source modules).

In the aspect of heat dissipation, the cold plate cooling method of non-conductive coolant is adopted.

The 51.2T silicon NPO cold plate liquid-cooled switch, with the same height, upgrades the NPO module from 1.6T to 3.2T, and the front panel supports 64 800G connectors, each of which can be divided into two 400G ports for forward compatibility. The number of external light source modules has increased to 16.

51.2T NPO cold plate liquid-cooled switch in the actual network, 51.2T NPO switch (released commercially at the end of 2023 at the earliest) can be used in 100G / 200G access network as access-convergence equipment to achieve high-speed interconnection.

It is worth mentioning that the technology and product research and development of NPO / CPO is not a simple thing, but a test of the overall R & D strength of an enterprise.

The fact that Ruijie Network was able to launch NPO / CPO products in the world is the result of their continuous investment in painstaking R & D and innovation, as well as their technological leadership in this field.

Ruijie Network began to pay attention to silicon technology in 2019 and formally established a R & D and product team in June 2020. As members of OIF / COBO, they have always participated in the global meetings of the working group and participated in the discussion and formulation of relevant standards.

OIF working group global meeting site in the direction of silicon, Ruijie network has been in the forefront of the world, the future can be expected.

█ conclusion

Well, with so many introductions, I believe you have seen clearly what NPO / CPO is.

These two technologies are undoubtedly the development direction of data center network equipment. In the current wave of digitization, our pursuit of computing power and network communication ability is endless. In the pursuit of performance, we should also strive to balance power consumption. After all, what we want to take is the path of sustainable development.

It is hoped that silicon optical technology, represented by NPO / CPO, can further accelerate the landing and contribute to the green and low-carbon information infrastructure.

What kind of technological innovation will silicon technology bring in the future? Let's wait and see!

-- end of the full text--

This article comes from the official account of Wechat: fresh Jujube classroom (ID:xzclasscom), author: Xiaozaojun

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.