Shulou(Shulou.com)06/03 Report--

one。 Overview

Strong self-healing ability is an important feature of container orchestration management engine such as Kubernetes. Typically, Kubernetes achieves self-healing by restarting the failed container. Besides, is there any other way to implement the health check of containers based on Kubernetes orchestration? Liveness and Readiness are good choices.

two。 Practical steps

2.1 default health check for the system.

ApiVersion: v1kind: Podmetadata: labels: test: healthcheck name: healthcheckspec: restartPolicy: OnFailure containers:-name: healthcheck image: busybox args:-/ bin/sh-- c-sleep 10

Create a yaml file with the same content as described above, named HealthCheck.yaml,apply:

[root@k8s-m health-check] # kubectl apply-f HealthCheck.yamlpod/healthcheck created [root@k8s-m health-check] # kubectl get podNAME READY STATUS RESTARTS AGEhealthcheck 0 Universe 1 CrashLoopBackOff 3 4m52s

As we can see, the pod is not working properly and has been rebooted three times. The specific restart log can be viewed through the describe command, and I won't repeat it here. Let's execute the following command:

[root@k8s-m health-check] # sh-c "sleep 2ten exit 1" [root@k8s-m health-check] # echo $? 1

We can see that the command executes normally and the return value is 1. By default, a return value of 0 after the execution of the Linux command indicates that the command was executed successfully. Because the return value after successful execution is not 0, Kubernetes defaults to container failure and keeps restarting. However, there are many situations where the service actually fails, but the process does not exit. In this case, restart is often a simple and effective means. For example, a 500 server internal error is displayed when accessing the web service, which can be caused by many reasons, and rebooting may fix the failure quickly.

2.2 in Kubernetes, you can tell kebernetes when to restart self-healing through a Liveness probe.

ApiVersion: v1kind: Podmetadata: labels: test: liveness name: livenessspec: restartPolicy: OnFailure containers:-name: liveness image: busybox args:-/ bin/sh-- c-touch / tmp/healthcheck;sleep 30; rm-rf / tmp/healthcheck;sleep 600 livenessProbe: exec: command:-cat-/ tmp/healthcheck initialDelaySeconds: 10 periodSeconds: 5

Create a file named Liveness.yaml and create a Pod:

[root@k8s-m health-check] # kubectl apply-f Liveness.yamlpod/liveness created [root@k8s-m health-check] # kubectl get podNAME READY STATUS RESTARTS AGEliveness 1 Running 1 5m50s

From the yaml file, we can see that the / tmp/healthcheck file is created after the container is started, deleted after 30s, and sleep the process for 600s after deletion. Use cat / tmp/healthcheck to detect whether the container is malfunctioning. If the file exists, the container is normal, and the file does not exist, then kill the container and restart.

InitialDelaySeconds:10 specifies that the probe is performed 10 seconds after the container starts. Generally, this value is greater than the startup time of the container. PeriodSeconds:5 indicates that a probe is performed every 5 seconds, and if three consecutive Liveness probes fail, the container will be killed and restarted.

2.3 Readiness can tell Kubenentes when the container can be added to Service's load balancer pool to provide services.

ApiVersion: v1kind: Podmetadata: labels: test: readiness name: readinessspec: restartPolicy: OnFailure containers:-name: readiness image: busybox args:-/ bin/sh-- c-touch / tmp/healthcheck;sleep 30; rm-rf / tmp/healthcheck;sleep 600 readinessProbe: exec: command:-cat-/ tmp/healthcheck initialDelaySeconds: 10 periodSeconds: 5

Apply this file:

[root@k8s-m health-check] # kubectl apply-f Readiness.yamlpod/readiness created [root@k8s-m health-check] # kubectl get podNAME READY STATUS RESTARTS AGEreadiness 0Universe 1 Running 0 84s [root@k8s-m health-check] # kubectl get podNAME READY STATUS RESTARTS AGEreadiness 0gamble 1 Completed 0 23m

As we can see from the yaml file, the configuration of Readiness and Liveness probes is basically the same and can be applied only with slight changes. Through kubectl get pod, we find that the main difference between the two Health Check lies in the second and third columns of the output. The third column of Readiness is always running, and the second column changes from 1 to 0 after a period of time. When the second column is 0 / 1, the container is not available. You can check it out with the following command:

[root@k8s-m health-check] # while true;do kubectl describe pod readiness;done

Liveness and Readiness are two Health Check mechanisms that are not interdependent and can be used at the same time.

three。 Development

3.1 the application of Health Check in Scale Up.

ApiVersion: apps/v1beta1kind: Deploymentmetadata: name: webspec: replicas: 3 template: metadata: labels: run: webspec: containers:-name: web image: httpd ports:-containerPort: 8080 readinessProbe: httpGet: scheme: HTTP path: / health-check port: 8080 initialDelaySeconds: 10 PeriodSeconds: 5---apiVersion: v1kind: Servicemetadata: name: web-svcspec: selector: run: web ports:-protocol: TCP port: 8080 targetPort: 80

With the above yaml, a service named web-svc and a Deployment named web are created.

[root@k8s-m health-check] # kubectl apply-f HealthCheck-web-deployment.yamldeployment.apps/web unchangedservice/web-svc created [root@k8s-m health-check] # kubectl get service web-svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT (S) AGEweb-svc ClusterIP 10.101.1.6 8080/TCP 2m20s [root@k8s-m health-check] # kubectl get deployment webNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGEweb 3 3 3 0 3m26s [root@k8s-m health-check] # kubectl get podNAME READY STATUS RESTARTS AGEweb-7d96585f7f-q5p4d 0/1 Running 0 3m35sweb-7d96585f7f-w6tqx 0/1 Running 0 3m35sweb-7d96585f7f-xrqwm 0/1 Running 0 3m35s

Focusing on lines 17-23, line 17 indicates that the Health Check mechanism used in this case is Readiness, and the probe method is httpGet. When Kubernetes judges the condition of successful detection of this method, the return value of http request is between 200,400. Schema specifies the protocol, which can be http (default) and https. Path specifies the access path and port specifies the port.

The container starts probing 10 seconds after startup. If the return value of http://container_ip:8080/health-check is not 200-400, the container is not ready and does not receive requests from Service web-svc. / health-check is the code we use to implement the probe. An example of the detection result is as follows:

[root@k8s-m health-check] # kubectl describe pod webWarning Unhealthy 57s (x219 over 19m) kubelet, k8s-n2 Readiness probe failed: Get http://10.244.2.61:8080/healthy: dial tcp 10.244.2.61 kubectl describe pod webWarning Unhealthy 8080: connect: connection refused

3.2 Application of Health Check in rolling update (Rolling Update).

By default, during the Rolling Update process, Kubernetes assumes that the container is ready and gradually replaces the old copy. If the container of the new version fails, the whole application may not be able to process the request and provide services after the version update is completed. If such incidents occur in a production environment, the consequences will be very serious. If the Health Check is configured correctly, only the new copy that passes the Readiness probe can be added to the Service. If it does not pass the probe, the existing copy will not be replaced and the business will still run normally.

Next, we create the yaml files app.v1.yaml and app.v2.yaml, respectively:

ApiVersion: apps/v1beta1kind: Deploymentmetadata: name: appspec: replicas: 8 template: metadata: labels: run: appspec: containers:-name: app image: busybox args:-/ bin/sh-- c-sleep 10 Touch / tmp/health-check Sleep 30000 readinessProbe: exec: command:-cat-/ tmp/health-check initialDelaySeconds: 10 periodSeconds: 5apiVersion: apps/v1beta1kind: Deploymentmetadata: name: appspec: replicas: 8 template: metadata: labels: run: appspec: containers:-name: app image: busybox args: -/ bin/sh-- c-sleep 3000 readinessProbe: exec: command:-cat-/ tmp/health-check initialDelaySeconds: 10 periodSeconds: 5

Apply file app.v1.yaml:

[root@k8s-m health-check] # kubectl apply-f app.v1.yaml-- recorddeployment.apps/app created [root@k8s-m health-check] # kubectl get podNAME READY STATUS RESTARTS AGEapp-844b9b5bf-9nnrb 1 Running 0 2m52sapp-844b9b5bf-b8tw2 1 Running 0 2m52sapp-844b9b5bf-j2n9c 1 2m52sapp-844b9b5bf-j2n9c 1 Running 0 2m52sapp-844b9b5bf-ml8c5 1 Running 0 2m52sapp-844b9b5bf-mtgr9 1/1 Running 0 2m52sapp-844b9b5bf-n4dn8 1/1 Running 0 2m52sapp-844b9b5bf-ppzh7 1/1 Running 0 2m52sapp-844b9b5bf-z55d4 1/1 Running 0 2m52s

Update to app.v2.yaml:

[root@k8s-m health-check] # kubectl apply-f app.v2.yaml-- recorddeployment.apps/app configured [root@k8s-m health-check] # kubectl get podNAME READY STATUS RESTARTS AGEapp-844b9b5bf-9nnrb 1 Running 0 3m30sapp-844b9b5bf-b8tw2 1 Running 0 3m30sapp-844b9b5bf-j2n9c 1 Running 0 3m30sapp-844b9b5bf-ml8c5 1/1 Terminating 0 3m30sapp-844b9b5bf-mtgr9 1/1 Running 0 3m30sapp-844b9b5bf-n4dn8 1/1 Running 0 3m30sapp-844b9b5bf-ppzh7 1/1 Terminating 0 3m30sapp-844b9b5bf-z55d4 1/1 Running 0 3m30sapp-cd49b84-bxvtc 0/1 ContainerCreating 0 6sapp-cd49b84-gkkj8 0/1 ContainerCreating 0 6sapp-cd49b84-jfzcm 0/1 ContainerCreating 0 6sapp-cd49b84-xl8ws 0/1 ContainerCreating 0 6s

We will observe later:

[root@k8s-m health-check] # kubectl get podNAME READY STATUS RESTARTS AGEapp-844b9b5bf-9nnrb 1 4m59sapp-844b9b5bf-b8tw2 1 Running 0 4m59sapp-844b9b5bf-b8tw2 1 Running 0 4m59sapp-844b9b5bf-j2n9c 1 4m59sapp-844b9b5bf-mtgr9 1 Running 0 4m59sapp-844b9b5bf-mtgr9 1 4m59sapp-844b9b5bf-n4dn8 0 4m59sapp-844b9b5bf-n4dn8 1 Running 0 4m59sappa- 844b9b5bf-z55d4 1/1 Running 0 4m59sapp-cd49b84-bxvtc 0/1 Running 0 95sapp-cd49b84-gkkj8 0/1 Running 0 95sapp-cd49b84-jfzcm 0/1 Running 0 95sapp-cd49b84-xl8ws 0/1 Running 0 95s

At this point, the status is all running, but there are still 4 Pod with a READY of 0apace 1. Take another look:

[root@k8s-m health-check] # kubectl get deployment appNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGEapp 8 10 4 6 7m20s

DESIRED said that the expected number of copies is 8 Magi current, that the current number of copies is 10, and that the number of upgraded copies is 4, and that the number of available replicas is 6. If no changes are made, this state will be maintained forever. Here, it is important to note that 2 old replicas are deleted and 4 new replicas are created in Rolling Update. Leave it here for the last discussion.

Version rollback to v1:

[root@k8s-m health-check] # kubectl rollout history deployment appdeployment.extensions/appREVISION CHANGE-CAUSE1 kubectl apply-- filename=app.v1.yaml-- record=true2 kubectl apply-- filename=app.v2.yaml-- record=true [root@k8s-m health-check] # kubectl rollout undo deployment app--to-revision=1deployment.extensions/app [root@k8s-m health-check] # kubectl get podNAME READY STATUS RESTARTS AGEapp-844b9b5bf-8qqhk 1 Running 0 2m37sapp-844b9b5bf-9nnrb 1/1 Running 0 18mapp-844b9b5bf-b8tw2 1/1 Running 0 18mapp-844b9b5bf-j2n9c 1/1 Running 0 18mapp-844b9b5bf-mtgr9 1/1 Running 0 18mapp-844b9b5bf-n4dn8 1/1 Running 0 18mapp-844b9b5bf-pqpm5 1/1 Running 0 2m37sapp-844b9b5bf-z55d4 1 Running 0 18m 4. Summary

Liveness and Readiness are two different ways of Health Check in Kubernetes. They are very similar, but different. You can use both at the same time, or you can use them alone. Specific differences have been mentioned above.

4.2 in my last article on Rolling Update, I mentioned replacement rules during rolling updates. In this article, we still use the default way to update. The maxSurge and maxUnavailable parameters determine the number of copies in each state during the update process, and the default values for both parameters are 25%. After the update, the total number of copies is 8-8-0.25-10, and the number of available copies is 8-8-0.25-6. During this process, 2 copies were destroyed and 4 new copies were created.

4.3 when going online in a general production environment, use Health Check as far as possible to ensure that the business is not affected. The realization means of this process are diversified, which need to be summarized and selected according to the actual situation.

five。 Related materials

5.1official documentation: about Liveness and Readiness

5.2official documentation: about maxSurge and maxUnavailable

5.3 the code involved in this article

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.