Shulou(Shulou.com)06/02 Report--

Tonight, the fifth issue of online training of 20Rod 30 Kubernetes Master Class "Log, Monitoring and Alerts in Kubernetes" will be broadcast soon. Click http://live.vhall.com/317569561 to book registration for free! Jieshao

Kubernetes is an open source container orchestration framework that provides us with a simple way to deploy, extend, and monitor. In this article, we will discuss the built-in monitoring capabilities of Kubernetes. In order to make it easier for readers to understand, this article includes some demonstrations.

Overview of Kubernetes architecture

At the infrastructure level, a Kubernetes cluster is a group of physical machines or virtual machines that perform specific functions. The physical or virtual machine that plays the primary role is responsible for the entire operation and coordinates container management running on all node.

The Master component manages the lifecycle of pod:

Apiserver: expose the main component of API for all other master components

Scheduler: responsible for determining which node pod should run on according to the information in the pod specification

Controller-manager: responsible for node management (detection of node failure), pod replication and endpoint creation

Etcd: key / value storage for storing all internal cluster data

The Node component is a worker machine managed by master in Kubernetes. Each node contains the necessary components to run pod:

Kubelet: handles all communication between master and the node running on it. It works with the container runtime and is responsible for deploying and monitoring containers

Kube-proxy: responsible for maintaining the network rules of node and handling communication between pod, node, and the outside world.

Container Runtime: runs the container on node.

Logically, a Kubernetes deployment is made up of components that play their respective roles in the cluster:

The basic deployment unit within the Pod:Kubernetes. An pod consists of one or more containers that share network namespaces and IP addresses.

Service: acts as a load balancer. They provide IP addresses before the pool (a set of pod) and also provide policies to control access to IP addresses.

ReplicaSet: it is controlled by deployment and is responsible for ensuring that the number of pod required by deployment is working properly.

Namespace (namespace): defines logical isolation for different types of resources, such as pod or service.

Metadata: Mark the container according to its deployment characteristics.

Monitoring Kubernetes

Monitoring applications is essential if we want to predict problems and identify potential bottlenecks in development or deployment.

To help monitor the cluster and many of the active components that make up the deployment, Kubernetes provides some built-in monitoring capabilities:

Kubernetes dashboard: provides an overview of the resources running on the cluster. It also provides a very basic deployment and a way to interact with these resources.

CAdvisor: an open source agent for monitoring resource usage and analyzing container performance.

Liveness and Readiness Probe: actively monitor the health of the container.

Horizontal Pod Autoscaler: increase the number of pod as needed, based on the information collected by analyzing different metrics.

In this article, we will focus on the first two built-in tools. In the next article in this series, we will introduce other monitoring tools.

There are many indicators in Kubernetes that need to be monitored. Just as we describe the architecture in two different ways (infrastructure and logic), we can also divide monitoring into two main components: monitoring the cluster itself and the workload monitoring running on the cluster.

Cluster monitoring

All clusters should monitor the underlying server components because server-tier problems tend to occur in workloads. Some metrics to pay attention to when monitoring node resources include CPU, disk, and network bandwidth. Knowing these metrics will let us know whether we need to expand or reduce the capacity of the cluster (this is especially important if the enterprise uses a cloud provider and attaches great importance to the operating cost).

Workload monitoring

We also need to consider metrics related to deployment and its pod. One of the important points is to compare the number of pod currently running in deployment with the expected number. In addition, we should also pay attention to health examination, container indicators and final application indicators.

Preparation in advance

In the following sections, we will introduce the built-in monitoring features listed one by one in the form of demo, and the preparations we need to do for this are:

Google Cloud platform account: you can use the free trial version. If you are using other mainstream cloud platforms, the method of operation is similar.

The host used to run Rancher: it can be a personal PC / Mac or a VM in the public cloud.

Google Cloud SDK: should be installed with kubectl on the host running Rancher. Make sure gcloud has access to your Google cloud account by using your credentials for authentication (gcloud init and gcloud auth login).

Start the Rancher instance

The first step is to start the Rancher instance. Rancher has a very intuitive getting started guide for reference: https://rancher.com/quick-start/

Deploy a GKE cluster using Rancher

Follow the instructions to set up and configure the Kubernetes cluster using Rancher:

Https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/gke/

Note: please make sure that Kubernetes Dashboard is enabled, and the Kubernetes version we use here is v.1.10.

Kubernetes Dashboard

Kubernetes dashboard is a Web-based Kubernetes user interface that we can use to troubleshoot applications and manage cluster resources.

Rancher, on the other hand, can help users install dashboard with one click. The main uses of dashboard include:

Provides an overview of cluster resources (including the overall situation and each node), showing all namespaces, and listing all defined storage classes

Show all applications running on the cluster

Provides information about the status of Kubernetes resources in the cluster and any errors that may occur

To access dashboard, we need to proxy requests between our computer and the Kubernetes API server. Enter the following code to start the proxy server using kubectl:

The proxy server starts in the background and outputs something similar to the following:

Now, to view dashboard, visit the following address through your browser:

Http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

Then, we need to enter the appropriate credentials on the login page:

In the future, we will learn how to use the service account mechanism to create users with administrator privileges. We will use two YAML files.

A YAML file is used to create a service account:

Another YAML file will create a ClusterRoleBinding for our user:

Apply two YAML files to create the objects they define:

After creating the user and setting the correct permissions, we need to find the token to log in:

Kubectl-n kube-system describe secret $(kubectl-n kube-system get secret | grep admin-user | awk'{print $1}')

Select "Token" in the Kubernetes dashboard credential prompt and enter the value you retrieved above in the field for authentication.

Kubernetes Dashboard contains several main views:

Administrative view: lists node, namespaces, and persistent volumes, and other details. We can get the node integration page (CPU and memory usage) as well as a separate detail page for each node, showing its metrics, specifications, status, allocated resources, and pod.

Workload view: shows all applications running in the selected namespace. Summarizes important information about the workload, such as the number of pod prepared in StatefulSet or deployment, or the current memory usage of pod.

Service Discovery and load balancing View: shows Kubernetes resources that expose services to the outside world and enables service discovery within the cluster.

Configuration and Storage View: shows the persistent volume declaration resources used by the application. The configuration view shows all the Kubernetes resources used for real-time configuration of applications running in the cluster.

Without any workload running, the dashboard page will be empty because nothing will be deployed on the Kubernetes at this time. If you want to browse all the views provided by dashboard, the best choice is to deploy applications that use different types of workloads (StatefulSet, deployment, replica sets, and so on). This article on how to deploy Redis on Kubernetes is a good example of how dashboard displays information when deploying a Redis cluster (a stateful set with volume declarations and configMaps) and a test application (a Kubernetes deployment).

After configuring the workload, we can close a node and check the different tabs to see some updates:

CAdvisor

CAdvisor is an open source agent integrated into kubelet binaries to monitor resource usage and analyze container performance. CAdvisor collects statistics about CPU, memory, file, and network usage of all containers running on a given node (cAdvisor does not run at the pod layer). In addition to the core metrics, cAdvisor monitors events. Users can access metrics directly using commands such as kubectl top, or they can use the scheduler to execute metrics at the scheduling layer (for example, using autoscaling).

It is important to note that cAdvisor does not store certain metrics for a long time, so if you need to use this feature, you should look for a dedicated monitoring tool.

Since Kubernetes version 1.10, cAdvisor's UI has been almost deprecated, and after Kubernetes 1.12, cAdvisor's UI will be completely deleted. Rancher allows you to choose the version of Kubernetes used for clustering. When setting up the infrastructure for this demonstration, we configured the cluster to use version 1.10, so we can still access cAdvisor UI.

To access cAdvisor UI, we need to proxy between our computer and the Kubernetes API server. Enter the following command to start the local instance of the proxy server:

Next, find the name of the node:

You can view UI in your browser at the following address and replace the node name with the identifier you found on the command line:

Http://localhost:8001/api/v1/nodes/gke-c-plnf4-default-pool-5eb56043-23p5:4194/proxy/containers/

To confirm that kubelet is listening on port 4194, you can log in to node for more information:

We can confirm that in our version of Kubernetes, the kubelet process provides cAdvisor Web UI through this port:

If you are running Kubernetes version 1.12 or higher, kubelet will no longer listen on port 4194 because cAdvisorUI has been deleted. You can use the above command to confirm. However, since cAdvisor is part of the kubelet binaries, the relevant metrics still exist.

The kubelet binary exposes all runtime and cAdvisor metrics using the Prometheus presentation format:

Http://localhost:8001/api/v1/nodes/gke-c-plnf4-default-pool-5eb56043-23p5/proxy/metrics/cadvisor

Among a lot of output, the indicators you can focus on are:

CPU:

Ocontainer_cpu_user_seconds_total: the cumulative time spent by the user on CPU in seconds

Ocontainer_cpu_system_seconds_total: cumulative time consumed by the "system" CPU in seconds

Ocontainer_cpu_usage_seconds_total: cumulative time consumed by CPU in seconds (sum of the above)

Memory:

Ocontainer_memory_cache: number of bytes of page cache memory

Ocontainer_memory_swap: container exchange usage in bytes

Ocontainer_memory_usage_bytes: current memory usage, in bytes, including all memory

Ocontainer_memory_max_usage_bytes: maximum memory usage in bytes

Disk:

Ocontainer_fs_io_time_seconds_total: the time in seconds it took to execute iThano

Ocontainer_fs_io_time_weighted_seconds_total: cumulative weighted Imax O time in seconds

Ocontainer_fs_writes_bytes_total: cumulative number of bytes written

Ocontainer_fs_reads_bytes_total: cumulative bytes read

Network:

Ocontainer_network_receive_bytes_total: cumulative number of bytes received

Ocontainer_network_receive_errors_total: cumulative number of errors encountered while receiving

Ocontainer_network_transmit_bytes_total: cumulative number of bytes transferred

Ocontainer_network_transmit_errors_total: cumulative number of errors encountered during transmission

Some other useful indicators:

/ healthz: the endpoint used to determine whether the cAdvisor is healthy

/ healthz / ping: check the connection to etcd

/ spec: returns the endpoint of cAdvisor MachineInfo ()

For example, to view cAdvisor MachineInfo (), we can visit:

Http://localhost:8001/api/v1/nodes/gke-c-plnf4-default-pool-5eb56043-23p5:10255/proxy/spec/

The pod endpoint provides the same output as kubectl get pods-o json for pod running on node:

Http://localhost:8001/api/v1/nodes/gke-c-plnf4-default-pool-5eb56043-23p5:10255/proxy/pods/

Similarly, you can get the log by visiting the following link:

Http://localhost:8001/logs/kube-apiserver.log

Conclusion

The importance of monitoring is self-evident, and it gives us a full understanding of the state of the application. Kubernetes has many built-in tools for users to choose from to better understand the infrastructure layer (node) and logical layer (pod).

In this article, we focus on tools that provide monitoring and metrics for users. In the next article in this series, we will continue to share monitoring tools that focus on workload scaling and lifecycle management.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.