Shulou(Shulou.com)06/01 Report--

This article is about how to update the application to achieve K8s interrupt rolling update, the editor feels very practical, so share with you to learn, I hope you can learn something after reading this article, do not say much, follow the editor to have a look.

In Kubernetes cluster, services are usually provided in the way of Deployment + LoadBalancer type Service, and the typical deployment architecture is shown in figure 1. The deployment and operation and maintenance of this architecture are very simple and convenient, but there may be service interruptions and online problems when applying updates or upgrades. Today we will analyze in detail why this architecture will cause service interruption when updating the application and how to avoid service interruption.

Figure 1 Business deployment diagram

Why is there a service outage?

When Deployment scrolls the update, it creates a new pod and waits for the new pod running before deleting the old pod.

Create a new Pod

Figure 2 schematic diagram of service interruption

Reason for interruption: after the Pod running is added to the Endpoint backend, CCS monitors the Endpoint change and adds the Node to the SLB backend. At this point, the request is forwarded from SLB to Pod, but the Pod service code has not been initialized and the request cannot be processed, resulting in service interruption, as shown in figure 2.

Solution: configure ready detection for pod and wait for the business code to be initialized before adding node to the SLB backend.

Delete Pod

In the process of deleting the old pod, multiple objects (such as Endpoint, ipvs/iptables, SLB) need to be synchronized in state, and these synchronization operations are performed asynchronously, and the overall synchronization process is shown in figure 3.

Fig. 3 Deployment update sequence diagram

Pod

Pod status change: set Pod to Terminating state and remove it from the Endpoints list of all Service. At this point, Pod stops getting new traffic, but containers running in Pod are not affected

Performing preStop Hook:Pod deletion triggers preStop Hook,preStop Hook to support bash script, TCP, or HTTP request

Send SIGTERM signal: send SIGTERM signal to the container in Pod

Wait for the specified time: the terminationGracePeriodSeconds field is used to control the wait time, and the default value is 30 seconds. This step is performed at the same time as preStop Hook, so terminationGracePeriodSeconds needs a longer time than preStop, otherwise pod will be kill before the execution of preStop is finished.

Send SIGKILL signal: after waiting for a specified time, send SIGKILL signal to the container in pod and delete pod.

Cause of interruption: the above steps 1, 2, 3 and 4 are carried out at the same time, so it is possible that the Pod has not been removed from the Endpoints after receiving the SIGTERM signal and stopping working. At this point, the request is forwarded from the slb to the pod, and the Pod has stopped working, so there is a service outage, as shown in figure 4.

Fig. 4 schematic diagram of service interruption

Solution: configure preStop Hook for the pod so that when the Pod receives the SIGTERM, the sleep stops working for a period of time instead of immediately, ensuring that traffic forwarded from the SLB can continue to be processed by the Pod.

Iptables/ipvs

Cause of interruption: when the pod changes to the termintaing state, the pod is removed from the endpoint of all service. Kube-proxy cleans up the corresponding iptables/ipvs entries. When the container service changes from watch to endpoint, slb openapi will be called to remove the backend, which will take several seconds. Because these two operations are performed at the same time, it is possible that the iptables/ipvs entry on the node has been cleaned but the node has not been removed from the slb. At this point, traffic flows in from slb, and there are no corresponding iptables/ipvs rules on the node that cause service interruption, as shown in figure 5.

Figure 5 schematic diagram of service interruption

Solution:

Cluster mode: in Cluster mode, kube-proxy writes all business Pod to the iptables/ipvs of Node. If there is no business pod in the current Node, the request will be forwarded to other Node, so there will be no service interruption, as shown in 6.

Figure 6 schematic diagram of Cluster mode request forwarding

Local mode: in Local mode, kube-proxy only writes pod on Node to iptables/ipvs. When there is only one pod on the Node and the state changes to terminating, iptables/ipvs removes the pod record. When the request is forwarded to this node, there is no corresponding iptables/ipvs record, which causes the request to fail. This problem can be avoided by upgrading in place, that is, ensuring that there is at least one Running Pod on the Node during the update process. Upgrading in place ensures that there will always be a business pod record in Node's iptables/ipvs, so there will be no service interruption, as shown in figure 7

Fig. 7 schematic diagram of request forwarding when Local mode is upgraded in place

ENI mode Service:ENI mode bypasses kube-proxy and mounts Pod directly to the SLB backend, so there is no service interruption caused by iptables/ipvs.

Figure 8 schematic diagram of ENI mode request forwarding

SLB

Figure 9 schematic diagram of service interruption

Reason for interruption: when the Endpoints is monitored by CCS, the Node will be removed from the slb backend. When the node is removed from the slb backend, the SLB will directly disconnect the persistent connection that continues to be sent to the node, resulting in service interruption.

Solution: set up long link elegant interrupts for SLB (depending on specific cloud vendors).

How to avoid service interruption

To avoid service interruption, we can start with two types of resources: Pod and Service. Next, we will introduce the corresponding configuration methods according to the reasons for the above interruption.

Pod configuration apiVersion: v1kind: Podmetadata: name: nginx namespace: defaultspec: containers:-name: nginx image: nginx # Survival Detection livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 30 successThreshold: 1 tcpSocket: port: 5084 timeoutSeconds: 1 # ready Detection readinessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 30 successThreshold: 1 tcpSocket: port : 5084 timeoutSeconds: 1 # elegant exit lifecycle: preStop: exec: command:-sleep-30 terminationGracePeriodSeconds: 60

Note: the detection frequency, delay time, unhealthy threshold and other data of ready Test (readinessProbe) need to be set reasonably. The startup time of some applications is long. If the setting time is too short, it will cause POD to restart repeatedly.

LivenessProbe is a survival test. If the number of failures reaches the threshold (failureThreshold), pod will restart. For more information on configuration, please see the official documentation.

ReadinessProbe is a readiness check, and pod will not be added to the Endpoint until the readiness check passes. The Endpoint will not be mounted to the slb backend until the CCS monitors the node change.

The preStop time is recommended to be set to the time it takes for the business to process all remaining requests, and the terminationGracePeriodSeconds time is recommended to be set to preStop plus more than 30 seconds.

Service configuration Cluster mode (externalTrafficPolicy: Cluster) apiVersion: v1kind: Servicemetadata: name: nginx namespace: defaultspec: externalTrafficPolicy: Cluster ports:-port: 80 protocol: TCP targetPort: 80 selector: run: nginx type: LoadBalancer

CCS mounts all nodes in the cluster to the backend of SLB (except those configured with BackendLabel tags), so it consumes SLB quota quickly. SLB limits the number of SLB that can be mounted on each ECS. The default is 50. When the quota is consumed, new listeners and SLB cannot be created.

In Cluster mode, if the current node does not have a business pod, it will forward the request to another Node. NAT is required when forwarding across nodes, so the source IP will be lost.

Local mode (externalTrafficPolicy: Local) apiVersion: v1kind: Servicemetadata: name: nginx namespace: defaultspec: externalTrafficPolicy: Local ports:-port: 80 protocol: TCP targetPort: 80 selector: run: nginx type: LoadBalancer# needs to allow each node to have at least one Running Pod# in the update process as much as possible by modifying UpdateStrategy and using nodeAffinity to ensure that Max Unavailable can be set to 0 in place rolling update# * UpdateStrategy Ensure that there is a new Pod to start before stopping the pod# * first apply label to a fixed number of nodes to schedule # * use nodeAffinity+ and the number of replicas that exceeds the number of relevant node to ensure that new Pod# is built in place as much as possible, for example: apiVersion: apps/v1kind: Deployment.strategy: rollingUpdate: maxSurge: 50% maxUnavailable: 0 type: RollingUpdate. Affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution:-weight: 1 preference: matchExpressions:-key: deploy operator: In values:-nginx

By default, CCS adds the node where the Pod corresponding to Service resides to the backend of SLB, so SLB quota consumption is slow. In Local mode, the request is forwarded directly to the node where the pod resides, and there is no cross-node forwarding, so the source IP address can be retained. In Local mode, service interruption can be avoided by upgrading in place, as shown in the yaml file above.

ENI mode (Aliyun specific mode) apiVersion: v1kind: Servicemetadata: annotations: service.beta.kubernetes.io/backend-type: "eni" name: nginxspec: ports:-name: http port: 30080 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer

In Terway network mode, set service.beta.kubernetes.io/backend-type:

The "eni" annotation can create a SLB for ENI mode. In ENI mode, pod is mounted directly to the SLB backend without going through kube-proxy, so there is no problem of service interruption. The request is forwarded directly to the pod, so the source IP address can be retained.

The comparison of the three svc modes is shown in the table below.

Figure 10 Service comparison

Conclusion Terway network model (recommended method)

Choose svc + in ENI mode to set Pod elegant termination + ready detection.

Flannel Network Mode  

If the number of slb in the cluster is small and there is no need to retain the source ip: choose cluster mode + set Pod elegant termination + ready detection

If there are a large number of slb in the cluster or need to retain the source ip: choose local mode + set Pod elegant termination + ready detection + in-place upgrade (ensure that there is at least one Running Pod on each node during the update process).

The above is how to update the application to achieve K8s interrupt rolling update, the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.