Shulou(Shulou.com)06/02 Report--

author| Shengdong Alibaba Cloud after-sales technical expert

Source: Docker, click to view the original text.

In my experience, understanding the concept of Kubernetes cluster services is not easy. Especially when we try to troubleshoot service-related problems based on plausible understanding, it can be very difficult.

This is reflected in the fact that for beginners, it is difficult to understand the basic problem of ping the IP address of the service; even for experienced engineers, it is quite challenging to understand the configuration of service-related iptables.

In today's article, I will explain in depth the principle and implementation of Kubernetes cluster services for everyone to understand.

What is the essence of Kubernetes cluster services

Conceptually, Kubernetes cluster services are actually Load Balancer, or Reverse Proxy. This has many similarities with Alibaba Cloud's Load Balancer product. Like Load Balancer, a service has its IP address and front-end port; it mounts multiple container group pods behind it as its "back-end servers," which have their own IP and listening ports.

When such a Load Balancer and backend architecture is combined with Kubernetes cluster, the most intuitive implementation we can think of is that one node in the cluster is dedicated to the role of Load Balancer (similar to LVS), while other nodes are used to load backend container groups.

There is a huge flaw in this implementation method, which is the single-point problem. Kubernetes cluster is the crystallization of Google's years of automated operations practices, which clearly deviates from its philosophy of intelligent operations.

Bring your own messenger

Sidecar is a core concept in the microservices domain. Sidecar mode, to put it another way, is to bring your own communicator. Those of you who are familiar with service grids will be familiar with this. However, few people may notice that the implementation of Kubernetes cluster original service is also based on Sidecar pattern.

In the Kubernetes cluster, the service implementation is essentially a reverse proxy Sidecar deployed for each cluster node. All access to cluster services is converted by the reverse proxy on the node into access to the service backend container group. Basically, the relationship between nodes and these sidecars is shown below.

Bring service to life.

In the previous two sections, we saw that the service of Kubernetes cluster is essentially a Load Balancer, i.e. a reverse proxy; at the same time, we know that in actual implementation, this reverse proxy is not deployed on a node of the cluster, but as a sidecar of cluster nodes, deployed on each node.

The real thing that lights the service into the reverse proxy here is a controller of the Kubernetes cluster, the kube-proxy. For more information on how Kubernetes cluster controllers work, please refer to my other article on controllers. In simple terms, kube-proxies act as controllers deployed on cluster nodes and listen for cluster state changes through the cluster API Server. When a new service is created, kube-proxy translates the state and attributes of the cluster service into the configuration of the reverse proxy.

The remaining problem is the reverse proxy, which is the implementation of Proxy in the above figure.

one implementation

Kubernetes cluster nodes implement service reverse proxy methods, there are three main methods, namely userspace, iptables and IPVS. Today we only analyze the way iptables are in depth, and the underlying network is based on Alibaba Cloud Flannel Cluster Network.

filter frame

Now, imagine a scenario. We have a room. The room has an inlet pipe and an outlet pipe. The water entering from the inlet pipe is not drinkable directly because of impurities. And we expect that the water that comes out of the outlet pipe can be drunk directly. To achieve this, we cut open the water pipe and add an impurity filter in the middle.

After a few days, our needs changed. We didn't just want the water coming out of the house to be drinkable, we wanted it to be hot. So we had to add another cut in the water pipe and then add a heater.

Obviously, this is an ugly way to cut open a pipe and add new functionality. Because demand can change at any time, it's hard to even guarantee that after a year and a half, this pipe will still find a place to cut open.

So we need to redesign it. First of all, we can't just cut the pipe, so we have to fix the cut. For example, in the case of the upper edge, we ensure that the water pipe can have only one cut position. Second, we abstract two ways of treating water: physical change and chemical change.

Based on the above design, if we need to filter impurities, we can add a rule for filtering impurities in the chemical change function module; if we need to increase the temperature, we can add a heating rule in the physical change function module.

The above filter frame is obviously much better than the way of cutting water pipes. In designing this framework, we mainly did two things, one was to fix the position of the water pipe incision, and the other was to abstract two ways of water treatment.

With these two things in mind, we can look at how iptables, or more accurately netfilter, works. Netfilter is actually a filter framework. netfilter cuts 5 ports on the network packet receiving and routing pipeline, namely PREROUTING, FORWARD, POSTROUTING, INPUT and OUTPUT; at the same time, netfilter defines several network packet processing methods including nat and filter.

It should be noted that routing and forwarding greatly increase the complexity of netfilter above. If we ignore routing and forwarding, netfilter will become as simple as our water filter framework.

node network large graph

Now let's take a look at the Kubernetes cluster node network overview. Viewed horizontally, the network environment on a node is partitioned into different network namespaces, including the host network namespace and the Pod network namespace; vertically, each network namespace includes the complete network stack, from applications to protocol stacks to network devices.

At the network device level, we build a virtual local area network inside the system through the cni0 virtual bridge. Pod networks are connected to this virtual local area network via veth pairs. The cni0 virtual local area network communicates with the outside world through host routing and port eth0.

At this layer of the network protocol stack, we can program the netfilter filter framework to implement reverse proxies for cluster nodes.

The implementation of reverse proxy, in the final analysis, is to do DNAT, that is, to send packets to the cluster service IP and port, modify the IP and port sent to a specific container group.

Referring to the netfilter framework diagram, we know that in netfilter, you can change the source address or destination address of a packet by adding NAT rules in PREROUTING, OUTPUT, and POSTROUGING.

Because what needs to be done here is DNAT, i.e. changing the destination address, which must occur before routing to ensure that the packet can be routed correctly, the rules for implementing reverse proxies need to be added to both PREROUTING and OUTPUT.

Among them, PREOURTING rules are used to handle traffic accessing services from pods. After the packet is sent from the Pod network veth to cni0, it enters the host protocol stack and is first processed through netfilter PREROUTING, so the packet sent to the service will be DNAT at this location. After DNAT processing, the destination address of the packet becomes the address of another Pod, which is then routed through the host to eth0 and sent to the correct cluster node.

The DNAT rule added at OUTPUT is used to process packets sent from the host network to the service. The principle is similar, that is, before routing, modify the destination address to facilitate routing.

Upgrade filter framework

In the Filter framework section, we see that netfilter is a filter framework. Netfilter cuts 5 ports on the data "pipe to" and does some packet processing work on these 5 ports respectively. Although fixing the notch locations and packet handling classification has greatly optimized the filter framework, there is a critical problem that we still have to modify the pipes to accommodate the new functionality. In other words, the framework does not completely decouple the plumbing and filtering functions.

In order to decouple the pipe and filter functions, netfilter uses the concept of tables. The table is the filtering center of netfilter. Its core function is the classification (table) of filtering methods and the organization (chain) of filtering rules in each filtering method.

After decoupling the filtering function from the pipeline, all processing of packets becomes configuration of tables. The five cuts on the pipeline only become the inlet and outlet of the flow, responsible for sending the flow to the filter center and continuing the flow after treatment along the pipeline.

As shown above, netfilter organizes rules into chains in the table. The table has default chains for each pipe cut, as well as custom chains that we added ourselves. Default chain is the entry point of data, default chain can jump to custom chain to complete some complex functions. The benefits of allowing custom chains to be added here are obvious. To accomplish a complex filtering function, such as implementing a reverse proxy for Kubernetes cluster nodes, we can modularize our rules using custom chains.

Reverse Proxy for Services with Custom Chains

The reverse proxy of cluster service actually implements DNAT translation of data packet modularly by using custom chain. KUBE-SERVICE is the entry chain of the entire reverse proxy, which corresponds to the total entry of all services;KUBE-SVC-XXXX chain is the entry chain of a specific service, and KUBE-SERVICE chain jumps to KUBE-SVC-XXXX chain of specific services according to service IP; KUBE-SEP-XXXX chain represents the address and port of a specific Pod, i.e. endpoint, and specific service chain KUBE-SVC-XXXX jumps to endpoint chain according to a certain algorithm (generally random).

As mentioned earlier, because what needs to be done here is DNAT, i.e. changing the destination address, this modification must occur before routing to ensure that the packet can be processed correctly by routing. So KUBE-SERVICE is called by the default chains PREROUTING and OUTPUT.

summary

Through this article, you should have a deeper understanding of the concept and implementation of Kubernetes cluster services. We basically need to grasp three main points:

A service is essentially a Load Balancer; a service Load Balancer is implemented using a Sidecar pattern similar to a service grid, rather than an LVS type exclusive pattern; and a kube-proxy is essentially a cluster controller. In addition to this, we thought about the design of the filter framework and, based on this, understood the principles of Load Balancer for services implemented using iptables.

Original link: yq.aliyun.com/articles/710873

"Alibaba Cloud Native Weixin Official Accounts (ID: Alicloudnative) focuses on micro-services, Serverless, containers, Service Mesh and other technical fields, focuses on cloud native popular technology trends, cloud native large-scale landing practice, and is the technical public account that best understands cloud native developers. "

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.