Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to determine when to restart the container by Kubernetes". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Elasticity is one of the most important factors to consider when designing mission-critical, highly available applications.

When an application can quickly recover from a failure, it is resilient.

Cloud native applications are typically designed to use a micro-service architecture, where each component is located in a container. To ensure that Kubernetes-hosted applications are highly available, there are specific patterns that need to be followed when designing clusters, including the "health detection mode." Applying the principle of High observability (HOP) ensures that every request received by your application finds a response in a timely manner.

The High Observability Principle (HOP)

The principle of high observability is one of the container-based application design principles. The microservice architecture requires that each service does not care (and should not care) how the callee handles the request.

The HOP principle requires that each service must expose several API endpoints. Its significance is to reveal the health status of the service, and Kubernetes invokes these endpoints to determine the next step of routing and load balancing.

"

Well-designed cloud native programs should log events to STDERR and STDOUT, and log intake services such as logstash and Fluent transport these logs to centralized monitoring (such as Prometheus) and log aggregation systems (such as ELK). The following figure illustrates how cloud native applications follow the health detection model and the principle of high observability.

How to Apply Health Probe Pattern in Kubernetes?

I have previously written the original ASP.NetCore + Docker health check: [web program exposes http health check endpoints, platform polling probe]. Kubernetes refines probes for different situations, and what is more powerful is to give corresponding decisions.

Liveness Probes

Use the Survival probe to determine when to restart the container.

Use survival probes to check whether the application itself is unresponsive and deadlocked, and sometimes restarting the container can often solve such problems.

Let's take Kubernetes' official demo as an example:

ApiVersion: v1

Kind: Pod

Metadata:

Labels:

Test: liveness

Name: liveness-exec

Spec:

Containers:

-name: liveness

Image: busybox

Args:

-/ bin/sh

-- c

-touch / tmp/healthy; sleep 30; rm-rf / tmp/healthy; sleep 600

LivenessProbe:

Exec:

Command:

-cat

-/ tmp/healthy

InitialDelaySeconds: 5 # instructs kubectl to wait 5 seconds before performing the first probe

PeriodSeconds: 5 # interval 5 seconds polling

The files in the container will be deleted 30 seconds after the first liveness probe of kubectl starts in 30 seconds, and then the probe with an interval of 5 seconds will fail. According to the default configuration of liveness, the probe will be abandoned if it fails three times in a row. Abandoning the probe means restarting the container. Therefore, the container will start the above process after restart in 45s, so you can see this probe trying to fix the application problem with the decision of restart.

This probe will be reflected in the RESTARTS column of kubectl get pod

Readiness Probes

Use the ready probe to determine whether the container is ready and whether it can accept traffic.

If all containers in the Pod ready, the Pod is considered as ready. If the pod does not have a ready, it will be removed from the service load balancer.

"

In some cases, when the application is temporarily unavailable (loading a large amount of data or relying on external services), it is useless to restart the Pod, but you also do not want the request to be sent to the Pod

The following applications are strongly dependent on mongodb, and we set readiness probes for these dependencies

Services.AddHealthChecks ()

.AddCheck (nameof (MongoHealthCheck), tags: new [] {"readyz"})

/ /--

App.UseHealthChecks ("/ readyz", new HealthCheckOptions)

{

Predicate = (check) = > check.Tags.Contains ("readyz")

});

The following code detects Mongodb connectivity

Sealed class MongoHealthCheck: IHealthCheck

{

Private readonly IMongoDatabase _ defaultMongoDatabase

Public MongoHealthCheck (IDefaultMongoDatabaseProvider defaultMongoDatabaseProvider)

{

_ defaultMongoDatabase = defaultMongoDatabaseProvider.GetDatabase ()

}

Public async Task CheckHealthAsync (HealthCheckContext context, CancellationToken cancellationToken = default)

{

Var doc = await _ defaultMongoDatabase.RunCommandAsync (

New BsonDocumentCommand (

New BsonDocument () {

{"ping", "1"}

})

CancellationToken: cancellationToken)

Var ok = doc ["ok"] .ToBoolean ()

If (ok)

{

Return HealthCheckResult.Healthy ("OK")

}

Return HealthCheckResult.Unhealthy ("NotOK")

}

}

For the detection of dependencies, the detection period and timeout can be set a little longer.

ReadinessProbe:

HttpGet:

Path: / readyz

Port: 80

InitialDelaySeconds: 5

PeriodSeconds: 60 # 60s probe once

TimeoutSeconds: 30 # detects a 30s timeout each time, which is the same as the connection timeout between application establishment and dependencies

FailureThreshold: 3 # failed to detect 3 times in a row, and the Pod will be marked as `Unready` Startup Probes

Use [startup probe] to determine whether the container application has been started. If this probe is configured, the survival and ready probes will be disabled until the probe succeeds.

Configure probe initialDelaySeconds: container starts, probe works later. Default is 0speriodSeconds probe detection cycle. Default 10stimeoutSeconds: timeout of probe work, default 1ssuccessThreshold: probe is considered successful several times in a row, default is failureThreshold: probe fails several times in a row, and the probe is considered to have failed in the end. For livenes probe, final failure means restart, and for readiness probe, the pod Unready is defaulted for 3 times.

It is strongly recommended to set the probe parameters reasonably according to the application structure to avoid frequent restarts or Unready caused by unrealistic failure.

That's all for "how Kubernetes determines when to restart the container". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.