Shulou(Shulou.com)05/31 Report--

This article introduces you how to understand the Kubernetes probe, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Configuring readiness, liveness, and startup probes can handle unhealthy Pod. The editor introduces three types of probes, best practices, and related tools to detect possible configuration problems.

One of the challenges of distributed systems and micro-service architectures is to automatically detect abnormal applications and reroute requests (request) to other available systems to recover damaged components. Health check-ups are a reliable way to meet this challenge. With Kubernetes, you can configure a health check with a probe to determine the status of each Pod.

By default, Kubernetes observes the Pod lifecycle and routes traffic to Pod when the container transitions from the pending state to the succeeded state. Kubelet monitors the crashed application and restarts Pod to recover. Many developers believe that such a basic setting is sufficient, especially if applications within Pod are also configured with daemon managers (such as Node.js 's PM2). However, there is an unexpected situation when Kubernetes thinks that Pod is healthy and can accept requests after all containers are started, but the application receives traffic before it is actually ready, such as initializing some state, establishing a database connection, or loading data before the application logic is processed. When the Deployment starts to scale, the application that is not ready receives traffic and returns a 500error, which creates a problem of the interval between the actual readiness of the application and what Kubernetes thinks it is.

Again, this is how the Kubernetes probe is used to define when the container is ready to accept traffic and when to restart the container. Since Kubernetes v1.16, three types of probes have been supported. These three types of probes, best practices, and related tools are introduced to detect possible configuration problems.

K8sMeetup

Kubernetes probe

Readiness and liveness probes are supported when Kubernetes version is less than v1.15. Startup probe is added as Alpha function in v1.16 and upgraded to Beta in v1.18.

All three probes have the following parameters: initialDelaySeconds: the number of seconds to wait before starting the liveness and readiness probes. PeriodSeconds: check the frequency of the probe. TimeoutSeconds: the number of seconds before the probe is marked as timed out (fails the health check). SuccessThreshold: the minimum number of consecutive successful checks that the probe needs to pass. FailureThreshold: the number of retries before marking the probe as failed. For liveness probes, this will cause Pod to restart. For the readiness probe, Pod is marked as not ready (unready). The Readiness probe readiness probe lets kubelet know when the application is ready to accept new traffic. If the application needs some time to initialize the state after the process starts, configure the readiness probe to make Kubernetes wait before sending new traffic. The main function of readiness probe is to direct traffic to deployment after service.

One important thing about the readiness probe is that it runs throughout the life cycle of the container. This means that the readiness probe runs not only at startup, but also repeatedly during the Pod run. This is to deal with situations where the application is temporarily unavailable (such as loading a large amount of data and waiting for an external connection). In this case, we don't have to kill the application, we can wait for it to recover. The readiness probe can be used to detect this situation and send traffic to these Pod after the Pod passes the readiness check again. Liveness probe liveness probe is used to restart unhealthy containers. Kubelet periodically ping liveness probes to determine health status and kill Pod if the liveness test fails. Liveness checking can help applications recover from deadlocks. If there is no liveness check, Kubernetes will think that the Pod in the deadlock is healthy, because from the Kubernetes's point of view, the Pod child process is still running and is healthy. By configuring the liveness probe, kubelet can detect that the application is in an unhealthy state and restart Pod to restore availability.

The Startup probe startup probe is similar to the readiness probe, but it is executed only at startup and can be optimized for containers that start slowly or applications that have unpredictable behavior during initialization. With the readiness probe, we can configure initialDelaySeconds to determine how long the readiness probe waits before it is ready.

Suppose you have an application that occasionally needs to download a large amount of data, and because initialDelaySeconds is a static number, we must set the most conservative wait time for the application, even if it does not require that long initialization wait time. With the startup probe, we can configure failureThreshold and periodSeconds to solve this problem, such as setting failureThreshold to 15 minutes seconds to 5, which means that the application will have startup time for 10x5=75s before it fails.

K8sMeetup

Configuration probe

Now that we know the different types of probes, here are three different ways to configure each probe.

HTTPkubelet sends the HTTP GET request to endpoint and checks the 2xx or 3xx response. We can reuse existing HTTP endpoint or set up a lightweight HTTP server for probing (for example, Express server with / healthz endpoint). The HTTP probe contains other additional parameters: host: the hostname to connect to (default: IP of pod). Scheme: HTTP (default) or HTTPS. Path: the path on the HTTP/S server. HttpHeaders: custom headers (if required for authentication, CORS settings, etc.). Port: the port name or port number of the access server.

TCP if you only need to check if a TCP connection can be established, you can specify a TCP probe. If a TCP connection is established, mark the Pod as healthy. HTTP probes may be useful for gRPC or FTP servers that are not suitable for TCP probes.

Command can configure the probe to run the shell command. If the exit code returned by the command is 0, the check passes, otherwise Pod will be marked as unhealthy. This type of probe is useful if you don't want to expose the HTTP server and port, or if you want to check the initialization steps with a command (for example, check whether a configuration file has been created, run the CLI command). K8sMeetup

Best practic

Although the exact parameters and usage of the probe depend on the application, there are some common best practices:

For older (≤ v1.15) Kubernetes clusters, the readiness probe with initial latency is used to handle the container startup phase. For newer (≥ v1.16) Kubernetes clusters, startup probes should be used for applications with unpredictable or variable startup times. The startup probe shares the same endpoint as the readiness and liveness probes (for example, / healthz), but the failureThreshold can be set higher than other probes to have a longer startup time, and the set failure time is more reasonable than liveness and readiness. If the readiness probe is not used for other signal purposes, the readiness and liveness probes can share the same endpoint, but if there is only one Pod (that is, using VPA), set the readiness probe to resolve the startup behavior and use the liveness probe to determine the health status. In this case, marking an unhealthy Pod means downtime. Readiness inspection can signal system failure in a variety of ways. For example, when an application loses its connection to the database, you can use the readiness probe to temporarily block new requests and allow the system to reconnect. It can also mark busy Pod as unprepared and balance the workload to other Pod.

In short, well-defined probes usually lead to better elasticity and usability. Be sure to observe the startup time and system behavior, and adjust the probe settings when the application changes.

K8sMeetup

Tools

Finally, in view of the importance of Kubernetes probes, we can use the Kubernetes resource analysis tool to detect missing probes. These tools can be run on an existing cluster or placed in a CI/CD process to automatically reject workloads without properly configuring resources.

Polaris: a resource analysis tool with a dashboard that can also be used as a validation webhook or CLI tool. Kube-score: a static code analysis tool for Helm, Kustomize, and standard YAML files. Popeye: a read-only utility for scanning Kubernetes clusters and reporting potential problems in the configuration. On how to understand the Kubernetes probe to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.