Shulou(Shulou.com)06/02 Report--

This article is to share with you what to do when there is an abnormal connection during the rolling update of the kubernetes service. The editor thinks it is very practical, so I share it with you. I hope you can get something after reading this article. Let's take a look at it.

During the rolling update of the service by kubernetes, some connections may be abnormal under the default configuration (for example, the connection is rejected). Let's analyze the reasons and give best practices.

Scroll to update the scene

Deploy services using deployment and associate service

Modify the replica of deployment to adjust the number of copies to scroll the update

Upgrade program version (modify image tag) triggers deployment to create a new replicaset to start a new version of pod

When you use HPA (HorizontalPodAutoscaler) to update the cause of abnormal connection in the deployment automatic expansion update process, the pod corresponding to service will be created or destroyed, that is, the endpoint list corresponding to service will be added or removed, and some connections may be abnormal during the update. The main reasons are:

Pod is created and added by endpoint controller to the endpoint list of service before it is fully started, and then kube-proxy configures the corresponding routing rule (iptables/ipvs). If the request is routed to the pod that has not been fully started, the pod cannot process the request properly, which will lead to a connection exception.

Pod is destroyed, but there is a time lag from endpoint controller watch to changing and updating service's endpoint list to kube-proxy updating routing rules. Pod may have been completely destroyed, but the routing rules have not been updated yet, resulting in requests still being forwarded to the destroyed pod ip, resulting in connection exception best practices.

For the first case, you can add a readinessProbe to the container in the pod, so that the container can be fully started before it is added to the service's endpoint list by endpoint controller, and then the kube-proxy updates the routing rules. In this case, all the back-end pod to which the request is forwarded are running normally, avoiding connection anomalies.

For the second case, you can add preStop hook to the container in the pod, so that the pod can wait for sleep for a period of time before it is really destroyed, and leave some time for endpoint controller and kube-proxy to clean up the endpoint and routing rules. During this period, the pod is in the Terminating state. If a request is forwarded to the destroyed pod before the routing rule update is complete, the request can still be processed normally, because it has not been really destroyed.

Best practices yaml example:

ApiVersion: extensions/v1beta1kind: Deploymentmetadata: name: nginxspec: replicas: 1 selector: matchLabels: component: nginx template: metadata: labels: component: nginxspec: containers:-name: nginx image: "nginx" ports:-name: http hostPort: 80 containerPort: 80 protocol: TCP readinessProbe: httpGet: Path: / healthz port: 80 httpHeaders:-name: X-Custom-Header value: Awesome initialDelaySeconds: 15 timeoutSeconds: 1 lifecycle: preStop: exec: command: ["/ bin/bash" "- c", "sleep 30"] above are kubernetes rolling updates to the service. What if a connection exception occurs? The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.