Shulou(Shulou.com)06/02 Report--

The sixth phase of online training of 20 Kubernetes 30 Kubernetes Master Class will be broadcast on Wednesday evening. Click on the link: http://live.vhall.com/847710932 to book registration for free!

The importance of monitoring is self-evident, and it gives us a full understanding of the state of the application. Kubernetes has many built-in tools for users to choose from to better understand the infrastructure layer (node) and logical layer (pod).

In the first article in this series, we focused on Dashboard and cAdvisor, tools designed to provide monitoring and metrics for users. In this article, we will continue to share monitoring tools that focus on workload scaling and lifecycle management: Probe (probe) and Horizontal Pod Autoscaler (HPA). Similarly, all presentations will be in the form of demo.

The preliminary preparation of Demo

In the previous article in this series, we demonstrated how to start Rancher instances and Kubernetes clusters. As we mentioned in the last article, Kubernetes comes with some built-in monitoring tools, including:

Kubernetes dashboard: provides an overview of the resources running on the cluster. It also provides a very basic deployment and a way to interact with these resources.

CAdvisor: an open source agent for monitoring resource usage and analyzing container performance.

Liveness and Readiness Probe: actively monitor the health of the container.

Horizontal Pod Autoscaler: increase the number of pod as needed, based on the information collected by analyzing different metrics.

In the previous article, we took an in-depth look at the first two components, Kubernetes Dashboard and cAdvisor, and in this article, we will continue to explore the last two tools: probes and HPA.

Probe (probe)

There are two kinds of health examinations: liveness and readiness.

The readiness probe lets Kubernetes know if the application is ready to serve the traffic. The Kubernetes allows the service to send traffic to the pod only if the probe is allowed to pass. If the probe fails, Kubernetes stops sending traffic to the Pod until it passes again.

Readiness probes are very useful when your application takes quite a long time to start. Even if the process has been started, the service will not work until the probe successfully passes. By default, Kubernetes will start sending traffic as soon as the process in the container starts, but in the case of a readiness probe, Kubernetes will not allow the service to route traffic until the application is fully started.

The liveness probe lets Kubernetes know if the application is running. If it is running, no action is taken. If the application is not running, Kubernetes deletes the pod and starts a new pod to replace the previous pod. Liveness probes are very useful when your application stops providing requests. Because the process is still running, by default, Kubernetes will continue to send requests to pod. With the liveness probe, Kubernetes will detect that the application is no longer providing the request and will restart pod.

For liveness and readiness checks, the following types of probes can be used:

Http: one of the most common custom probes. Kubernetes ping a path that marks the pod as healthy if it gets a http response in the range of 200,300.

Command: when using this probe, Kubernetes will run the command in one of the pod containers. If the command returns the exit code 0, the container is marked as healthy.

Tcp:Kubernetes will attempt to establish an TCP connection on the designated port. If a connection can be established, the container is marked as healthy.

When configuring the probe, you can provide the following parameters:

InitialDelaySeconds: the time to wait before sending the readiness/liveness probe when the container is first started. For liveness checking, be sure to start the probe only after the application is ready, otherwise your application will continue to restart.

PeriodSeconds: the frequency at which the probe is executed (default is 10).

TimeoutSeconds: the timeout of the probe in seconds (default is 1).

SuccessThreshold: the minimum number of consecutive successful checks of the probe.

FailureThreshold: the number of times the probe failed before abandoning. Abandoning the liveness probe causes Kubernetes to restart pod. For liveness probes, pod will be marked as not ready.

Demonstration of Readiness probe

In this section, we will use command checks to configure the readiness probe. We will deploy two copies using the default nginx container. No traffic is sent to pod until a file named / tmp/healthy is found in the container.

First, create a readiness.yaml file by entering the following command:

ApiVersion: apps/v1kind: Deploymentmetadata: name: readiness-demospec: selector: matchLabels: app: nginx replicas: 2 template: metadata: labels: app: nginx spec: containers:-image: nginx name: nginx ports:-containerPort: 80 readinessProbe: exec: command:-ls- / tmp/healthy initialDelaySeconds: 5 periodSeconds: 5-apiVersion: v1kind: Servicemetadata: name: lbspec: type: LoadBalancer ports:-port: 80 protocol: TCP targetPort: 80 selector: app: nginx

Next, apply the YAML file:

We will see the deployment and services that are being created:

Pod will not enter the READY state unless the readiness probe passes. In this case, since there is no file named / tmp/healthy, it will be marked as a failure and the service will not send any traffic.

To better understand, we will modify the default nginx index pages of two pod. When requested, the first one will display 1 as a response, and the second will display 2 as a response.

Replace the specific pod name with the pod name created by the deployment on the computer:

Create the required files in the first pod to transition to the READY state and route traffic there:

The probe runs every 5 seconds, so we may have to wait a while to see the results:

Once the state changes, we can start clicking on the external IP of the load balancer:

Here we should see our modified Nginx page, which consists of a numeric identifier:

Creating a file for the second pod also causes the pod to enter the READY state. Traffic here will also be redirected:

When the second pod is marked READY, the service sends traffic to the two pod:

The output at this point should indicate that traffic is being distributed between the two pod:

Demonstration of Liveness probe

In this section, we will demonstrate the use of tcp to check the configured liveness probe. As mentioned above, we will deploy two copies using the default nginx container. If port 80 within the container is not listening, traffic will not be sent to the container and the container will be restarted.

First, let's take a look at the liveness probe demo:

ApiVersion: apps/v1kind: Deploymentmetadata: name: liveness-demospec: selector: matchLabels: app: nginx replicas: 2 template: metadata: labels: app: nginx spec: containers:-image: nginx name: nginx ports:-containerPort: 80 livenessProbe: tcpSocket: port: 80 initialDelaySeconds: 15 periodSeconds: 5- -- apiVersion: v1kind: Servicemetadata: name: lbspec: type: LoadBalancer ports:-port: 80 protocol: TCP targetPort: 80 selector: app: nginx

We can apply YAML with one command:

We can then examine the pod and modify the default Nginx page as above to use a simple 1 or 2 to represent the response.

First, find the name that Nginx deployed to pod:

Next, replace the default index page in each pod with a numeric identifier:

Traffic has been redirected by the service, so responses can be obtained immediately from the two pod:

Similarly, the response should indicate that traffic is being distributed between the two pod:

We are now ready to stop the Nginx process in the first pod to see the running liveness probe. Once Kubernetes notices that the container is no longer listening on port 80, the state of the pod will change and restart. We can observe some of the states of the transition until it works properly again.

First, stop the Web server process in one of the pod:

Now, when Kubernetes notices that the probe failed and takes steps to restart pod, review the status of pod:

You may see that pod undergoes a variety of state transitions before being healthy again:

If we go through our service request page, we will see the correct response from the second pod, the modified identifier "2". However, the pod you just created returns the default Nginx page from the container image:

This indicates that Kubernetes has deployed a completely new pod to replace the previously failed pod.

Horizontal Pod Autoscaler

Horizontal Pod Autoscaler (HPA) is a feature of Kubernetes that enables us to automatically scale down the number of pod required to deploy, replicate controllers, or replica sets based on observed metrics. In practical use, CPU indicators are usually the main trigger factor, but custom indicators can also be trigger factors.

Based on the measured resource usage, each part of the process is automated and does not require human intervention. Relevant metrics can be obtained from API such as metrics.k8s.io, custom.metrics.k8s.io or external.metrics.k8s.io.

In the following examples, we will demonstrate based on CPU metrics. The command we can use in this case is kubectl top pods, which displays the CPU and memory usage of pod.

First, create a YAML file that will use a single copy to create the deployment:

Enter the following to apply the deployment:

This is a simple deployment with the same Nginx image and a single copy:

Next, let's learn how to implement the automatic scaling mechanism. Using the kubectl get / describe hpa command, you can view the currently defined autoscaler. To define a new autoscaler, we can use the kubectl create command. However, the easiest way to create an autoscaler is to target an existing deployment, as follows:

This will create an autoscaler for the hpa-demo deployment we created earlier, with an estimated CPU utilization of 50%. The replica number is set here to a number between 1 and 10, so the maximum number of pod that autoscaler will create when the load is high is 10.

You can confirm the configuration of autoscaler by entering the following:

We can also define it in YAML format to make it easier to review and change management:

To see how HPA is running, we need to run a command that creates a load on CPU. There are many ways, but a very simple example is as follows:

First, check the load on a unique pod. Because it is currently idle, there is not much load:

Now, let's generate some load on the current pod. Once the load increases, we should see that HPA starts automatically creating some additional pod to handle the increased load. Let the following command run for a few seconds, then stop the command:

Check the current load on the current pod:

HPA comes into play and starts to create additional pod. Kubernetes shows that the deployment has been automatically scaled, and there are now three copies:

We can see the details of HPA and the reasons for extending it to three copies:

Name: hpa-demoNamespace: defaultLabels: Annotations: kubectl.kubernetes.io/last-applied-configuration= {"apiVersion": "autoscaling/v1", "kind": "HorizontalPodAutoscaler" "metadata": {"annotations": {}, "name": "hpa-demo", "namespace": "default"}, "spec": {"maxRepli...CreationTimestamp: Sat" 30 Mar 2019 17:43:50 + 0200Reference: Deployment/hpa-demoMetrics: (current / target) resource cpu on pods (as a percentage of request): 104% (104m) / 50%Min replicas: 1Max replicas: 10Conditions: Type Status Reason Message-AbleToScale True ReadyForNewScale recommended size matches current size ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited False DesiredWithinRange the Desired count is within the acceptable rangeEvents: Type Reason Age From Message-Normal SuccessfulRescale 15s horizontal-pod-autoscaler New size: 3 Reason: cpu resource utilization (percentage of request) above target

Because we stopped the command to generate the load, HPA should notice the load reduction and reduce the number of copies if we wait a few minutes. Without a high load, there is no need to create two additional pod.

The default time for autoscaler to wait before performing a reduce operation in Kubernetes is 5 minutes. You can modify this time by adjusting the-horizontal-pod-autoscaler-downscale-delay setting. For more information, please refer to the official documentation:

Https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

When the wait time is over, we will find that the number of deployed pod decreases at the high-load tags:

The number of pod changes back to the cardinality:

If we check the description of HPA again, we will see the reason for the reduction in the number of copies:

Name: hpa-demoNamespace: defaultLabels: Annotations: kubectl.kubernetes.io/last-applied-configuration= {"apiVersion": "autoscaling/v1", "kind": "HorizontalPodAutoscaler" "metadata": {"annotations": {}, "name": "hpa-demo", "namespace": "default"}, "spec": {"maxRepli...CreationTimestamp: Sat" 30 Mar 2019 17:43:50 + 0200Reference: Deployment/hpa-demoMetrics: (current / target) resource cpu on pods (as a percentage of request): 0 / 50%Min replicas: 1Max replicas: 10Conditions: Type Status Reason Message-AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 1 ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rateEvents: Type Reason Age From Message-Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 3 Reason: cpu resource utilization (percentage of request) above target Normal SuccessfulRescale 13s horizontal-pod-autoscaler New size: 1; reason: All metrics below target

Conclusion

I believe that through these two articles, we can gain an in-depth understanding of how Kubernetes uses built-in tools to set up monitoring for clusters. We know how Kubernetes works continuously behind the scenes to keep the application running, and we should learn more about the principles behind it if we can.

Collecting all the data from Dashboard and probes, and exposing all container resources through cAdvisor, can help us understand resource limitations or capacity planning. Good monitoring of Kubernetes is critical because it helps us understand the health and performance of the cluster and the applications running on the cluster.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.