Shulou(Shulou.com)06/02 Report--

K8S uses ready and survival probes to configure health check. In order to solve this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a simpler and easier way.

Health examination

Health check (Health Check) can be used to monitor the status of services, such as D Monitoring of DNSPOD under Tencent. It is actually a health check to configure an access path to determine whether a website can be accessed normally. When a health check fails, an email notification or SMS will be sent to inform the webmaster to repair.

K8S traffic forwarding

In some modern distributed systems, user access is no longer a single host, but a cluster of hundreds of instances, and user requests are distributed to different instances through load balancers. Load balancing helps to solve the access pressure of a single server, while improving the high availability of the system, and health check is often used as a criterion to judge whether the current instance is "available". That is, when the system finds that the health check of an instance fails, the load balancer will not direct traffic to the instance.

Nowadays, cloud service providers such as AWS are generally equipped with health check for load balancers, while Kubernetes provides two probes, Liveliness and Readiness, to check the status of the container. According to official documents, Liveliness probe is to check whether the container is running, translated as survival probe (livenessProbe), and Readiness probe is to check whether the container is ready to accept HTTP requests, translated as ready probe (readinessProbe).

In Kubernetes, Pod is the smallest deployable computing unit that Kubernetes creates and manages. A Pod consists of one or more containers (Docker,rocket, etc.) that share memory, network, and how the container is run.

Survival probes and ready probes in the context of Kubernetes are called health checks. These container probes are small processes that run periodically, and the results returned by these probes (success, failure, or unknown) reflect the state of the container in the Kubernetes. Based on these results, Kubernetes determines how to handle each container to ensure flexibility, high availability, and longer uptime.

Ready probe

The ready probe is designed to let Kubernetes know if your application is ready to service the request. Kubernetes forwards traffic to Pod only if the probe is ready to pass. If the ready probe detection fails, Kubernetes stops sending traffic to the container until it passes.

Survival probe

The Liveness detector is to let Kubernetes know if your application is alive. If your app is still alive, then Kubernetes keeps it alive. If your application is dead, Kubernetes will remove Pod and restart one to replace it.

Working process

Let's look at two scenarios to see how ready probes and survival probes can help us build a more available system.

Ready probe

An application often takes a period of time to warm up and start, for example, the startup of a back-end project needs to connect to the database to perform database migration, etc., and the startup of a Spring project also depends on the Java virtual machine. Even if the process is started, your service cannot run until it is up and running. The application should not receive traffic until it is fully ready, but by default, Kubernetes starts sending traffic as soon as the process in the container starts. Probe through the ready probe until the application is fully started before allowing traffic to be sent to the new copy.

The working process of the ready probe

Survival probe

Let's imagine another situation in which our application fails to respond to user requests due to some reason "downtime" or deadlock situation after successful startup.

By default, Kubernetes continues to send requests to Pod and detects it by using a survival probe. When it is found that the service cannot process the request within a limited time (request error or timeout), the problematic pod will be restarted.

The working process of survival probe

Probe type

Probe type refers to how to carry out health check. There are three types of probes for K8S: HTTP,Command and TCP.

HTTP

HTTP probes are probably the most common type of probe. Even if the application is not a HTTP service, you can create a lightweight HTTP server to respond to probes. For example, if you let Kubernetes access a URL through HTTP, if the return code is in the range of 200 to 300, mark the application as healthy, otherwise it will be marked as unhealthy.

For more information on HTTP detection, please refer to here.

Command

For command probe, Kubernetes runs commands within the container. If the command returns with the exit code 0, the container is marked as normal. Otherwise, it is marked as unhealthy.

For more information about command detection, please refer to here.

TCP

The last type of probe is TCP probe, where Kubernetes attempts to establish an TCP connection on a designated port. If it can establish a connection, the container is considered healthy; if it cannot be considered unhealthy. This is often used to probe gRPC or FTP services.

For more information on TCP detection, please refer to here.

Initial detection delay

We can configure how often the K8S health check runs, check the conditions for success or failure, and the response timeout. You can refer to the documentation for configuring probes.

Failure to detect the survival probe will cause the pod to restart, so it is important to configure the initial probe delay initialDelaySeconds to ensure that the probe starts after the application is ready. Otherwise, the application will restart indefinitely!

I suggest using the P99 startup time as the initialDelaySeconds, or take the average startup time plus a buffer. At the same time, update this value according to the startup time of the application.

Give an example

Take the following configuration code for K8S as an example

K8S will use HTTP to access the / actuator/health of port 8080 120 seconds (initialDelaySeconds) after the start of Pod. If it exceeds 10 seconds or the return code is not within 200 seconds 300, the readiness check will fail.

Similarly, while Pod is running, K8S still detects / actuator/health at port 8080 every 5s (periodSeconds detects port 8080)

ApiVersion: apps/v1beta1kind: Deployment. ReadinessProbe: httpGet: path: / actuator/health port: 8080 initialDelaySeconds: 120 timeoutSeconds: 10 livenessProbe: httpGet: path: / actuator/health port: 8080 initialDelaySeconds: 60 timeoutSeconds: 10 periodSeconds: 5 this is the answer to the question about what is the health check for K8S use-ready and survival probe configuration. I hope the above content can help you to a certain extent, if you still have a lot of doubts to be solved, you can follow the industry information channel to learn more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.