Shulou(Shulou.com)06/02 Report--

Introduction:

A few years ago, when Boyun started the research and development of its own container network, in addition to the consideration of technology selection, we also had an in-depth discussion on whether to do Underlay or Overlay network first. At that time, most of the open source community and mainstream container manufacturers were mainly based on Overlay network solutions, but after an in-depth understanding of the real needs of many customers, we found that some customers had a very strong demand for direct connection between inside and outside the container. After much deliberation, we decided to do the Underlay network first (and then Overlay). With the development of the industry and the company itself, we have built and implemented more and more projects, which makes us think more and more deeply about the container network, and the point of view becomes clearer and clearer:

The Underlay network with internal and external direct connection is the correct way to open the container network.

01

Based on the requirements, consider the container network scheme.

As shown in the figure above, this is a typical scenario of enterprise micro-service containerization deployment, and it is also the direct reason that drives us to build Underlay network:

The service is deployed within the Kubernetes cluster (figure: service 1, service 2)

Database, registry, Redis, MQ and other components are deployed outside the cluster

Some services may also be deployed outside the cluster, such as the container platform trial phase (figure: service 3)

These services and components need to be directly interconnected.

If you need to meet the above requirements, the most simple and effective solution is to directly connect the Kubernetes internal and external network, that is, to adopt the Underlay network mode.

Of course, there are other ways to solve the above needs. For example, detailed analysis of the traffic between services and components, using ingress, egress, including rewriting application code, and so on, which can also be used under certain circumstances. However, on the one hand, these are specific solutions under specific scenarios, which lack a certain degree of versatility; on the other hand, they are prone to problems such as complex configuration, introduction of additional risks, difficult to locate errors and so on. It is not as simple and effective as directly connecting the internal and external networks through the Underlay network.

Another way is to put all services and components within the Kubernetes cluster, but this solution is still not common for specific scenarios, and it is difficult to ensure that all enterprise applications are in a Kubernetes cluster.

Similarly, the following scenarios are required for direct interconnection between internal and external networks of the container:

Specific-purpose Kubernetes clusters provide services. For example, a Kubernetes cluster dedicated to providing PaaS middleware services, a Kubernetes cluster dedicated to providing CI/CD services, a Kubernetes cluster dedicated to providing big data services, etc.

Service / component interconnection across multiple clusters

In order to reduce the risk in the pilot phase, some services run in the cluster and some services run outside the cluster.

02

Compared with virtual machines, the container network scheme is considered from the nature of the container.

In the process of container application practice, in addition to the application scenario, we also think about the container continuously from the perspective of underlying infrastructure.

From the perspective of infrastructure, containers and virtual machines are essentially the same, and they are the host infrastructure for upper-layer applications.

Therefore, from the bottom point of view, the demand for container network is the same as that of virtual machines. So, what is the landing mode of virtual machine network?

As shown in the right side of the figure above, the landing network solution of the IaAS layer can be considered to be an industry standard, whether it is based on VMware or OpenStack, basically using OVS (or similar to OVS) layer 2 Underlay network scheme.

Therefore, from the perspective of infrastructure, the container network adopts a scheme similar to IaaS, that is, the virtual machine and the container are placed on the same network level, which is the most reasonable choice.

PS: on public clouds, virtual machines are all in VPC, so the current container network solution for public clouds is mainly based on putting containers and virtual machines in the same VPC, which can be directly interconnected. This is also a typical proof of the above judgment.

At the same time, for the problem that some customers reported that the Unberlay network occupies too many IP addresses, we can also get a reasonable explanation from the point of view of the comparison between the virtual machine and the container: if the virtual machine is used for deployment, the number of IP addresses occupied is the same as the container Underlay network. The number of IP addresses is determined by the number of applications, and the use of containers does not introduce excess IP addresses. In addition, Ipv6 has begun to hit the ground on a large scale, and in the Ipv6 era, the number of IP addresses will not be a problem.

03

Technical scheme selection

Typical open source Unberlay network selection solutions for containers are Calico and MACVLAN, and the problems of these two solutions are also obvious:

Calico: the BGP routing protocol needs to be turned on in the data center router (or layer 3 switch), while BGP is the routing protocol of the WAN, which usually does not start inside the data center, and the support for low-end layer 3 switch / router alignment is also risky.

MACVLAN: some customers adopted this container network solution a few years ago. The biggest problem with MACVLAN is that the community activity has been very low, and some problems have not been solved in the community for a long time. At the same time, the future-oriented scalability is also poor.

These are the reasons why Boyun developed its own Underlay (also supports Overlay) network based on OVS.

04

Summary

In the container network scheme, the Overlay network scheme has the characteristics of low requirements for the underlying network (the landing process does not need to deal with the network department), easy landing, less IP address occupation and so on. However, as more and more customers apply Kubernetes and containers to the production environment on a large scale, the proportion of Bo Yun customers choosing to use Underlay network mode is also increasing. This makes us realize more clearly:

The Underlay network with internal and external direct connection is the correct way to open the container network.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.