Shulou(Shulou.com)06/02 Report--

This article is reproduced from the official account "Goose Factory Network Affairs" (ID:tencent_network) on Wechat. The author: Huang Xiang.

Goose Factory is an Internet company with a wide range of business types, covering large social flow platforms (Wechat / QQ), online games, public cloud, media (news / video), mobile applications, open platforms, Internet finance, etc., with different types of business having their own technology application characteristics, performance goals and cost consideration systems, resulting in different demands for backstage technology support capabilities. As an important part of the infrastructure, the network is also faced with various challenges brought by these massive business operations. Fortunately, Goose Factory is an ecosystem with strong professional and technical capabilities and smooth internal cooperation. With "constantly improving user experience" as a unified technical goal, many things can be accomplished through distributed cooperation. Based on this characteristic, as the architect of the goose factory basic network platform is also very happy, because it can be more convergent and more focused to solve some common key problems-- standardized basic network platform.

How to build this huge basic network platform itself is a very complex issue, in which network technology itself may account for less than 20% of the weight. in order to form a healthy system for network planning, construction and operation, and constantly improve the user-oriented experience of the business system. There are a lot of issues to consider, including network technology selection, company resource distribution trend, end-to-end operation system, network technology development trend, hardware supply chain management, comprehensive cost control, Trade-off between iterative and stable operation, wind power, domestic / international communication environment, and so on. This article focuses on the network itself, and the author strives to use an easy-to-understand description to simply share the idea of building a basic network platform in Goose Factory.

First, take a look at the overall architecture diagram of the goose factory basic network platform (below). The reason why the network has evolved to what it is today is mainly because the value subject of the goose factory network is two capabilities that are constantly built and optimized-- "the ability to connect services and services" and "the ability to connect services and users". The development of network architecture must evolve around two capabilities. At the same time, according to the characteristics of upper-level business (offline / online), the richness of geographical resources (ground / electricity), ISP network layout, disaster preparedness requirements, comprehensive cost composition and other factors, the overall basic network platform is divided into three parts:

Data Center: a data center network that provides high-speed switching capabilities to all servers within the same campus.

Edge: edge network, the border network used to connect subscribers (ISP).

DCI: the data center interconnected wide area network is responsible for connecting these Data Center scattered around the world to the Edge.

Each network architecture will follow the value goal of the overall basic network, and evolve and optimize with its own unique characteristics. The following discusses the construction ideas of these three major network architectures.

Data center network

The data center network focuses on the "ability to connect services and services" and occupies a high weight in the whole network investment sector. through years of accumulation, the data center network has formed tens of thousands of network elements. In the face of such a massive network, there must be a strict construction system to support, including design, construction, operation, supply chain management and other links, because any mistake means unimaginable scope of influence and rework. According to common sense, the data center network is the closest network to the business, and such a large volume should be the most stressful part of the operation. in fact, this problem is not as serious as imagined in the goose factory. Goose factory is an ecosystem with strong professional ability and smooth internal cooperation. after years of running-in, the upper business and basic network have formed a good tacit understanding of cooperation. The robustness, disaster recovery design and scheduling capabilities of the business system architecture (especially platform-level businesses) have reached a very high level, so that the architects of the basic network platform can use standardized and robust technologies to meet almost most of the business requirements for the data center network, so that they can have more energy to focus on the lower level and more professional areas related to the data center network.

With regard to the play of the goose factory data center network, or what capabilities are required to run the data center network, the experience and ideas of these years can be summarized as follows:

Data center network and infrastructure environment (Campus/Building) are deeply integrated for overall design and delivery, and multi-level CLOS scheme is adopted for the overall end-to-end design of Campus-level/Building-level, including equipment power consumption planning, equipment layout, wiring planning, physical fault domain planning, etc., in order to achieve the optimal solution of comprehensive architecture, construction, cost and maintenance. In order to make an in-depth study of the CLOS structure of the data center network from the perspective of methodology (the above is the logical diagram of the data center network of Family G in the Sigcomm paper), this topic is actually not as simple as imagined. It is a set of methodology system that comprehensively considers a series of problems, such as switch cost, optical cost, wind, thermal and hydropower environment, network technology, etc., about how to build a CLOS network. The Goose Factory architect even summed up a whole set of formula algorithms, which will be discussed in the future.

It is necessary to have a more in-depth understanding of the switched network architecture, which is the most basic technology reserve part of network architecture design. Whether self-developed switches or commercial switches, there are higher requirements for technology development, testing capabilities, and mastery of basic technologies, including switching chips, optical components, system protocol stacks, accumulation of SDK use, and so on.

The overall analysis ability of the current situation and trend of the industry, the overall control ability of the industry supply chain, and can have the agile ability to adapt to the changes of the industry environment. For example, with the continuous improvement of the server access rate, the proportion of light in the total cost of the data center network is getting higher and higher. We should be able to gain insight into these trends and changes in advance, and adjust the strategy and architecture according to our own situation.

Massive standardized data center network production has become a routine work throughout the year, which requires a set of online automatic management tool systems throughout the life cycle of architecture design, construction, operation, expansion, assets, change, retirement, etc., in order to ensure the healthy operation of IDC production business. Goose Factory reconstructed a set of such tool systems in 2014 and continuously optimized iterations, from architecture design to computer room retirement. Can be done online.

The operation tool platform needs to monitor the data center network from multiple dimensions, including white-box monitoring methods, such as alarm management and convergence for network elements themselves, and black-box monitoring schemes, such as Full-Mesh 's Probe to reflect the health of the network. It also includes mapping and interaction with upper-level business. For example, Wechat, the largest traffic platform in Goose Factory, has a large number of servers and a complete set of network quality monitoring system. It cooperates with the network platform to quickly find faults and isolate linkage operations.

The correct use of SDN methodology is to build the working content and effectiveness of SDN system according to the scenario. SDN focuses more on Routing adaptability, centralized link state maintenance, congestion management, fault shielding and other aspects of large-scale network, rather than SDN for ALL.

Edge Network (overseas)

Edge Network (Edge) focuses on "the ability to connect services and users". Its main task is to deliver goose factory services to users all over the world with the shortest path and the best quality. The domestic edge network generally exists in the form of static docking ISP, while overseas is dominated by BGP docking. This chapter mainly introduces the overseas edge network architecture of Goose Factory.

Due to the large number of overseas ISP, the global Internet is a very complex network environment. If all the services of Goose Factory are sent directly from overseas Data Center to several local big ISP to reach every user in the world, it is very difficult to provide a very good network experience for users everywhere. The goose factory made great efforts to solve this problem:

The goose factory has deployed a lot of resources to obtain the detection and quality data of services from users around the world, which can be used as the basis for business development and network acceleration, so that targeted architecture plans and construction plans can be developed.

The edge network is used to connect each ISP, which we call "egress". There are two types of egress architecturally: Region-based main egress Edge and Edge-POP for area network acceleration.

Edge, as the main exit based on Region, is often close to Data Center and connects a large number of ISP as the default exit of the Region.

As the coverage point of regional acceleration, the rhythm of Edge-POP planning and construction is considered according to many factors, including the number of local Internet users, upper-level business planning, coverage quality, local communication environment, comprehensive cost and so on.

All Edge and Edge-POP can be regarded as a whole resource pool, hosted on the DCI network.

Edge network will also encounter many technical challenges, such as multi-exit management ability, traffic scheduling ability, fault recovery ability and so on. In the early days, traditional network methods were used to manage multiple exits and schedule traffic. Often because of the deterioration of the quality of one exit, the goose factory network engineer needed to manually log on to the network equipment and use scripts to adjust the routing strategy. To pull traffic to better quality exits, as the number of exits continues to increase, whether in terms of network planning or manual optimization of traffic Are becoming more and more complex and powerless. To take a simple example, when there are only two exits, planning and optimization are very simple, either double active or active / standby, turn off whoever has a problem, and the dynamic routing protocol will automatically converge, which seems very simple. but imagine how to plan the rules for the use of these exits when there are hundreds or even thousands of exits? Under normal circumstances, what services or user traffic does each exit take? What is the backup relationship between these exits in case of failure or quality deterioration? Considering and designing comprehensively such factors as bandwidth, cost, and internal load of the interconnected ISP network, this has become a very complex issue. Once upon a time, there were thousands of routing policy commands on the export network equipment of the goose factory, some related to planning, some related to optimization, some related to handling failures, the network operation became more and more complex, and the number of exports was still growing rapidly.

A few years ago, we realized that if we did not restructure the design of this piece, sooner or later, we would not be able to play. Just as the tide of SDN was sweeping in, we borrowed the idea of SDN and spent a long time to build the capability of this piece, forming a very important competitiveness of the goose factory network. The core of this capability is the centralized control of multiple exports. We internally call it "choosing exits and scheduling traffic from God's point of view". To put it simply, we collect or input all exit information such as bandwidth, routing, traffic, quality, cost, and the corresponding relationship between IP and AS into the central control system, and then develop an algorithm that meets our business requirements to achieve centralized computing, so as to ensure that users from all ISP and all over the country can visit the goose factory with the best network quality under the current infrastructure conditions. After the calculation is completed, the execution policy is sent to the forwarding device to pull the traffic to the ground. At present, this complete system has been launched on the existing network, and at the same time, we have also built a set of service layer on this platform, so that upper-level businesses can develop APP and use this set of centralized control system to meet their needs, such as DDOS nearby cleaning and one-click blocking, public cloud customer traffic automatically switching exits, platform-level business area quality optimization, and so on.

DCI wide area network

The DCI wide area network is also responsible for building "the ability to connect services and services" and "the ability to connect services and users", that is, the network that connects all Data Center in Goose Factory around the world, which is the top priority of the entire basic network platform. The above-mentioned multi-exit traffic scheduling (traffic between overseas DC and overseas users) is also carried on this WAN. The picture above is a schematic diagram of the DCI network architecture. It can be noticed that there are two wide area networks in the middle. This design is closely related to the upper business characteristics of the goose factory. At first, there is only one DCI network, that is, DCI for Elastic Services, which carries all the wide area traffic and is an overloaded network. a few years ago, we began to use TE technology to improve the utilization and traffic scheduling capacity of this network, making the technical iteration and construction and expansion of this network a frequent daily task, while the fault tolerance and architecture design of goose factory platform-level services are very strong. Cooperation with the basic network is also more tacit understanding, so that this network can develop at a high speed. In recent years, financial business and public cloud business have gradually become the focus of network protection. These two services are quite different from the previous platform-level services with large traffic, and put forward extremely high requirements for network quality and availability. In order to cope with the changes in the company's business, we began to build a second network (DCI for Interactive Services in the figure) to focus on serving this part of the business. The characteristics of the two networks are described as follows:

DCI for Elastic Services: high-traffic wide area network, which carries more than 90% of the wide area traffic of goose factory and serves the mature platform-level business of most goose factories. It is a network with high frequency of rapid iteration and expansion of new technologies, and some link utilization sometimes reaches more than 80%.

DCI for Interactive Services: high-quality wide area network, service Internet finance and other services that are highly dependent on network quality, are built with general, mature and stable technologies, link utilization is controlled below 40%, there are no new technology iterations, and the frequency of capacity expansion is also very low.

As mentioned in the previous article, network architecture design is a huge system engineering, which needs to consider a lot of factors, and the most basic of these factors, as well as the closest part of the network itself is networking and technology selection. Wan topology is built on demand, and with the development of business and the growth of traffic, a very complex disordered topology is finally formed. There is a significant difference compared with the networking of data centers. The core reasons are probably as follows:

The distributed architecture of Internet companies is popular, namely Network as a Computer, and the network has become a part of the business system, which requires high bandwidth to support the flexible construction of the business system, while the bandwidth in the data center is cheap, the wide area network bandwidth is expensive and the delay is high, so the high bandwidth requirements and strong coupling business modules are concentrated in the data center, and the calls and bandwidth usage between business modules across the WAN will be much more cautious. This leads to the general phenomenon of high bandwidth per server in the data center and low bandwidth per server across the wide area network.

For the network, the bandwidth means the cost, which is mainly composed of the network element hardware cost and the link cost. The hardware cost of the data center network element accounts for most of the cost, and the link cost is low, while the WAN hardware cost is small, and the link cost is very high.

Combining the above two points, the data center network is usually built with low-cost, high-bandwidth, simple network equipment, namely "Fast and Stupid Fabric", to obtain higher bandwidth at the lowest cost. On the other hand, the bandwidth construction and expansion of the WAN will be very cautious, basically according to the capacity usage to expand the capacity on demand, resulting in a disordered topology.

With regard to traffic scheduling, point-to-point bandwidth management within the data center is relatively extensive, because the bandwidth is easy to obtain, and the characteristics of network elements are simple. CLOS architecture is used to heap high bandwidth, and each access destination has only one direction (ECMP is regarded as one direction), so there is no need for scheduling at all. On the other hand, the WAN topology is out of order, and there are many non-equivalent paths to a destination. When the access relationship is very large and the traffic is carried in an unordered topology, traffic scheduling is inevitable.

The above four points basically describe the characteristics of the wide area network, which are the most basic elements to be considered when designing the architecture, so the following focuses on the construction idea of the wide area network "DCI for Elastic Services" in Goose Factory.

Wide area links are extremely expensive, and the proportion of network element hardware costs is very low, so the crux of the problem is how to improve link utilization on the premise of obtaining the best network quality. Apart from the network, technology is one of the most critical factors in any field that wants to improve the utilization of resources, so we have made great efforts in the technical application of the wide area network. Just imagine, if you use the traditional routing method to drive this wide area network, you will only get one result, that is, the overall utilization of the network is not high, but local congestion often occurs, and the optimization team is expanding its capacity every day. This result is ironic. The reason for this problem is that traditional routing methods are not intelligent and perceptual enough to perceive where resources can be used, and the technical method we choose to solve this problem is "centralized traffic engineering system". The main idea is described as follows:

Decoupling "routing control system" and "path control system", routing is related to business, can only determine where the destination is, the path is related to the direction of traffic, and can determine which link can be taken between source and destination. After the routing system calculates the destination, it is handed over to the routing system to find the best path and send it to the destination accurately. This idea is the same as the real-time navigation of the Internet that we usually use.

Because the routing control system only needs to know the destination and does not care about how to get to the destination, the overall logic is very simple, so we use the traditional BGP to deliver the route, which is not SDN, stable and efficient.

The path control system is much more complex, for example, when we use navigation, the destination only needs to be input. the specific path needs to judge and calculate the best path according to many factors, such as the shortest distance, the traffic jam on each road, the number of traffic lights, whether there is a traffic restriction number and so on. Therefore, the path control system also needs a "God perspective", which needs to uniformly consider the global topology, link load, delay, and even link cost, and get the final result after reasonable calculation, so this part needs to use the idea of SDN to solve the problem. The Goose Factory's approach is centralized path control, and the controller receives all the information needed by the whole network, and calculates centrally, and finally obtains a certain number of point-to-point Tunnel for the routing control system, and the controller should be aware of network failures and traffic changes in real time to optimize the best path of the whole network to ensure that all access traffic can obtain the best network quality in real time.

Traditional RSVP-TE is a similar solution, but it is slightly overloaded and complex, including interoperability problems across hardware platforms, high complexity and high cost of network element equipment, and the pressure and risk of RSVP message volume caused by too many Tunnel. All these bring some challenges to large-scale TE deployment, but the biggest problem is that all paths are head-end node computing, and different requirements seize resources from each other, which can not achieve the overall optimization of the whole network. That is, there is no "God's perspective".

In addition, differentiated services are also deployed on this wide area network, that is, traffic classification, and different levels of traffic enjoy different levels of service. when serious failures occur, high-level traffic gives priority to ensuring the best network quality. low-level traffic may be preempted for bandwidth and may detour or even lose packets. This is also one of the core technologies of the WAN, which can not only protect the key traffic, but also improve the overall resource utilization to a high level, forming a good balance between reliability and utilization.

Summary

The construction of a large-scale basic network platform is a complex system engineering that requires team patience and willpower, and requires a very strong planning and design ability, but more importantly, in the process of operation, combined with the changes in business planning, industrial chain, communication environment, principal contradictions, changes in the composition of comprehensive costs and other factors, can be agile follow-up and adjustment. This article covers a wide range of content, the author uses less space to talk about the details, while more content focuses on some ideas and experience when the goose factory is doing these work, hoping to bring you a little reference value.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.