Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about what the micro-service monitoring system based on Kubernetes is like. Many people may not know much about it. In order to make you understand better, the editor summarized the following content for you. I hope you can get something according to this article.

The monitoring system is the most important part of the operation and maintenance system and even the whole software product life cycle. Perfect monitoring can help us to find the fault in time in advance and quickly trace the location problem afterwards. In the cloud native architecture system represented by micro-services, the system is divided into multiple levels, the call links between services are complex, and there are many targets in the system that need to be monitored. Without a perfect monitoring system, it is difficult to ensure the stability of the overall service.

Monitoring objects and layering

In the actual scene, according to the monitoring object and system hierarchy, the monitoring system can be divided into basic layer, middle layer, application layer, business layer and so on. Details can be shown as shown in the figure:

The basic layer monitoring is to monitor the host server (including host, container) and its underlying resources to ensure the stable operation of the basic environment on which the application depends. There are two main directions for basic layer monitoring:

Resource utilization: it monitors hardware resources such as Istroke O utilization, CPU utilization, memory utilization, disk utilization, network load and so on. Avoid service failures caused by exhaustion of hardware resources caused by the application itself or other special circumstances.

Network communication: is to monitor the network status between servers. Network communication is an important cornerstone of the Internet. if network problems such as excessive delay and high packet loss rate appear in the network between hosts, it will seriously affect the business.

It should be noted that in the new native cloud infrastructure based on Kubernetes containerization technology, the monitoring of the basic layer should not only monitor the host itself, but also monitor the status of the Kubernetes cluster and the usage of container resources. This will be described in detail later when we build a basic layer monitoring system based on Kubernetes.

Middle-tier monitoring mainly refers to the monitoring of middleware software that application services such as Nginx, Redis, MySQL, RocketMQ, Kafka and other application services rely on, and their stability is also the key to ensure the continuous availability of applications. Generally speaking, specific middleware software will build a targeted monitoring system according to its own characteristics.

Application layer monitoring here refers to the monitoring of business service programs. Generally speaking, our focus on application monitoring is mainly reflected in the following aspects:

The HTTP interface requests access. Including interface response time, throughput, etc.

JVM monitoring metrics. For Java services, we will also focus on JVM performance-related metrics such as GC time, number of threads, FGC/YGC time, etc.

Resource consumption. After deployment, the application consumes certain resources, such as memory and CPU consumption.

The health status of the service. For example, whether the current service survives, whether it runs stably, and so on.

Invoke the link. In the micro-service architecture, due to the longer call link, it is also necessary to focus on monitoring the invocation relationship and invocation between services, so as to avoid system-wide avalanches caused by link failures between local upstream and downstream services.

Business layer monitoring is also an important concern of the monitoring system, in the actual scenario, if you just let the application run stably, it must be far from enough. Therefore, we often monitor the data generated by specific business, such as PV, UV and other parameters concerned by the website system, while we pay attention to the order quantity and success rate of backend systems such as transactions.

Business indicators are also the core elements that reflect the stability of the system. If something goes wrong with any system, it must be the business indicators that will be affected first. The setting of core business indicators varies with specific business and scenarios, so the monitoring of the business layer needs to build a business monitoring system with business characteristics.

Common types of monitoring indicators

In the index monitoring system, the operation of the whole system can be perceptively recognized through statistical indicators. When a problem occurs, the indicators will fluctuate first, and these fluctuations will reflect what went wrong with the system, so that the cause of the problem can be investigated accordingly. Below, let's take a look at the types of statistical indicators and the common statistical indicators, which is the basis for us to further understand the monitoring system of indicators.

On the whole, the common Metrics index types are: counter (Counter), measuring instrument (Gauge), histogram (Histogram), summary (Summary) these four types. Their characteristics are as follows:

1. Counter (Counter)

Counter is a type of indicator with cumulative characteristics, usually this value is Double or Long type. For example, the values of common statistical indicators such as QPS, TPS and so on are calculated in the form of counters and then calculated with some statistical functions.

two。 Measuring instrument (Gauge)

Represents the measurement of a numerical value at a certain point in time. Both the measuring instrument and the counter can be used to query the value of fixed content at a certain point in time, but unlike the counter, the value of the measuring instrument can be changed at will, can be increased or decreased. For example, to obtain the number of active threads in the Java thread pool, we use the getActiveCount method in ThreadPoolExecutor; in addition, there are more common CPU utilization, memory usage and other specific indicators are obtained through the measuring instrument.

3. Histogram (Histogram)

A histogram is a data structure that aggregates multiple values together and can represent the distribution of the data. For example, if you take a common response time Bucket, you can divide the response time data into multiple buckets, each representing a time interval, such as zero 100 milliseconds, 100 seconds 500 milliseconds, and so on. In this way, we can visually see the time distribution of requests over a period of time, which will help us to understand the time distribution.

4. Summary (Summary)

The summary is similar to the histogram, it also represents the data results over a period of time, but the data content of the summary is not the same. Generally used to identify quantile value, quantile value is actually what we often say TP90, TP99 and so on. For example, if there are 100 time-consuming values, arrange all the values from the lowest to the highest, and take the position of 90%. The value of this position is the value of TP90. If the corresponding value of this bucket is assumed to be 80ms, it means that all requests less than or equal to 90% are ≤ 80ms.

Kubernetes micro-service monitoring system

Previously, we described the layering of the monitoring system as a whole and several common types of indicators that need to be mastered by the index monitoring system. Next, we focus on the micro-service monitoring system based on Kubernetes.

From the perspective of monitoring objects and system layering, the monitoring system needs to monitor a wide range of areas, but from the perspective of micro-service monitoring, if your micro-service deployment is entirely based on the Kubernetes cloud native environment, then the main monitoring objects we need to pay attention to are the Kubernetes cluster itself and the micro-service application containers running in it. For example, the monitoring of container resource usage, such as CPU utilization, memory utilization, network, IUniver O, and other indicators.

Of course, this does not mean that we do not need to pay attention to the monitoring of physical machines, virtual machine devices or middle-tier software in the basic layer, but this part of the work is generally maintained by special personnel. If we are using cloud services, then most of the cloud service providers have provided monitoring support for us. In addition, the monitoring of the basic physical layer and most of the intermediate software is not the focus of this article, so we no longer do too much practice, we can have an overall understanding of it.

Back to the micro-service monitoring system with Kubernetes as the carrier, although the monitoring system of Kubernetes project was very complex, there were many solutions in the community. However, with the development of this system today, it has completely evolved into a unified scheme with the Prometheus project as the core. In this section, we will demonstrate how to build a micro-service monitoring system for Kubernetes around Prometheus.

1. Brief introduction to Prometheus

After years of practice and precipitation in the industry, the current monitoring system can be divided into four categories: 1), Metrics (metrics) monitoring based on time series; 2) Tracing (link) monitoring based on call chain; 3) monitoring based on Logging (log); 4), health check (Healthcheck). Among the above monitoring methods, Metrics monitoring is one of the most important monitoring methods.

Simply understand the form of Metrics, that is, the numerical points [Time,Value] generated at discrete points in time, and a group of [Time,Value] numerical point series composed of some index is also called time series, so Metrics monitoring is often called time series monitoring.

As mentioned above, we briefly describe the basic characteristics of the index system, and the next Prometheus is an open source Metrics monitoring system based on time series, which can easily store, query and alarm statistical indicators. The overall system structure of Prometheus is shown in the following figure:

As can be seen from the figure above, the core of Prometheus's work is to use Pull (pull) mode to collect Metrics data (monitoring metric data) of monitored objects, and then the Prometheus server aggregates the received metric data and stores it in TSDB (time series database, such as OpenTSDB, InfluxDB), so that it can be retrieved freely according to time.

With this core mechanism, the remaining components of Prometheus are mainly used to run with this mechanism. PushGateway, for example, allows monitored objects to push Metrics data to Prometheus as Push. On the other hand, Alertmanager can flexibly set alarm according to Metrics information.

In addition, Prometheus also provides a complete set of PromQL query language. Through its HTTP query interface, users can easily combine index data with Grafana (visual monitoring indicator display tool), thus flexibly customize the key indicator monitoring Dashboard (Kanban) belonging to the system itself.

2. Prometheus Operator installation and deployment

Before we briefly introduced the basic principles of Prometheus monitoring system, the following content will demonstrate how to use Prometheus to build a set of micro-service monitoring system for Kubernetes cluster.

In the actual application scenario, the deployment methods for different monitoring objects Prometheus will be different. For example, the object to be monitored is the underlying physical machine, or the database and other middleware systems deployed by the physical machine, then in this case, the deployment environment of the Prometheus monitoring system will generally be placed under the physical machine.

If it is for the monitoring of Kubernetes clusters, then the mainstream way is to deploy Promethues to Kubernetes clusters using Promethues-Operator, so that the monitoring of Kubernetes clusters and containers can be implemented in a more native way. Promethues-Operator here refers to the Promethues package specifically for Kubernetes, which can simplify the deployment and configuration of Promethues.

Next, let's demonstrate how to quickly install and deploy Promethues in Kubernetes through Promethues-Operator (for Kubernetes experimental environment, please refer to this column). The specific steps are as follows:

1), install Helm

During this installation, Kubernetes's package management tool Helm will be used. Helm is a package management tool for Kubernetes, similar to Maven in Java, Npm in NodeJs, and apt in Ubuntu and yum in CentOS. It is mainly used to simplify the deployment and management of applications by Kubernetes.

First, download the corresponding Helm installation package from Github. The specific commands are as follows:

# find the Helm-related release package in Github, and refer to the following link: https://github.com/helm/helm/releases # after determining the relevant version, download the specific installation version to a node with kubectl installed, wget https://get.helm.sh/helm-v3.4.0-rc.1-linux-amd64.tar.gz

Extract and copy the downloaded executable Helm file to the folder / usr/local/bin, as follows:

Tar-zxvf helm-v3.4.0-rc.1-linux-amd64.tar.gz # copy the downloaded executable helm file to the folder / usr/local/bin mv linux-amd64/helm / usr/local/bin/

Then execute helm version. If you can see the Helm version information, it means that the Helm client has been installed successfully, as shown below:

$helm version version.BuildInfo {Version: "v3.4.0-rc.1", GitCommit: "7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState: "clean", GoVersion: "go1.14.10"}

After installing the Helm client, since some public Kubernetes packages are managed in the remote repository, you also need to add the official repository of helm charts (Kubernetes installation package in Helm, also known as charts), as follows:

$helm repo add stable https://charts.helm.sh/stable

To check whether the local helm repository has been added successfully, the command is as follows:

$helm repo list NAME URL stable https://charts.helm.sh/stable

At this point, you can see the charts list of various components by viewing the Helm repository. The command effect is as follows:

$helm search repo stable NAME CHART VERSION APP VERSION DESCRIPTION stable/acs-engine-autoscaler 2.1.3 2.1.1 Scales worker nodes within agent pools stable/aerospike...

As shown above, various stable versions of the Kubernetes installation package can be viewed through the "helm search" command at this time!

2), Helm search for Prometheus-Operator installation package

Before installing Prometheus-Operator, we first use the "helm" command to search for Prometheus-related charts packages, as follows:

$helm search repo prometheus

The specific search results are shown below:

As shown in the figure above, we can see that the installation package of "stable/prometheus-operator" with version 0.38.1 can be searched in the Helm repository. Then you can install it through helm!

3) install Prometheus-Operator monitoring system in Helm

Next, install the prometheus-operator monitoring system through Helm, with the following command:

# create k8s namespace kubectl create ns monitoring # install promethues-operator monitoring system helm install promethues-operator stable/prometheus-operator-n monitoring through helm

After executing the installation command, the output is as follows:

WARNING: This chart is deprecated manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" NAME: promethues-operator LAST DEPLOYED: Mon Oct 26 10:15:45 2020 NAMESPACE: monitoring STATUS: deployed REVISION: 1 NOTES: * DEPRECATED * stable/prometheus-operator chart is deprecated. * Further development has moved to https://github.com/prometheus-community/helm-charts * The chart has been renamed kube-prometheus-stack to more clearly reflect * that it installs the `kube- prometheus` project stack, within which Prometheus * Operator is only one component. The Prometheus Operator has been installed. Check its status by running: kubectl-- namespace monitoring get pods-l "release=promethues-operator" Visit https://github.com/coreos/prometheus-operator for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

After executing the installation command, view the specific Kubernetes Pods information as follows:

$kubectl get po-n monitoring NAME READY STATUS RESTARTS AGE alertmanager-promethues-operator-promet-alertmanager-0 2 5m31s promethues-operator-grafana-5df74d9cb4 2 Running 0 5m42s prometheus-promethues-operator-promet-prometheus-0 3 Running 3 Running 1 5m31s promethues-operator-grafana-5df74d9cb4-5d475 2 Running 0 6m53s promethues-operator-kube-state-metrics-89d8c459f-449k4 1/1 Running 0 6m53s promethues-operator-promet-operator-79f8b5f7ff-pfpbl 2/2 Running 0 6m53s promethues-operator-prometheus-node-exporter-6ll4z 1/1 Running 0 6m53s promethues-operator-prometheus-node-exporter-bvdb4 1/1 Running 0 6m53s

As shown above, you can see that the components related to the Prometheus monitoring system are running in the Kubernetes cluster as Pod.

Demonstration of Prometheus monitoring effect

Through the previous actual operation, we have deployed the Prometheus Operator installation package in the Kubernetes cluster through Helm. At this time, Prometheus has actually begun to play a role, and collected the operation index information of all kinds of Kubernetes. You can view this through Promethues's built-in monitoring interface. The specific steps are as follows:

Check the Pod node where the built-in monitoring interface resides in Kubernetes, and the command is as follows:

Kubectl-n monitoring get svc

Use nodeport to expose the promethues-operator built-in interface service outside the cluster, and specify port 30444 as follows:

Kubectl patch svc promethues-operator-promet-prometheus-n monitoring-p'{"spec": {"type": "NodePort", "ports": [{"port": 9090, "targetPort": 9090, "nodePort": 30444}]} 'service/promethues-operator-promet-prometheus patched

Enter the IP+ port address of the host where the Pod node is located in the browser. The URL example is as follows:

Http://10.211.55.11:30444/graph

At this point, you can see the built-in monitoring visual interface of Promethues, as shown in the following figure:

If you view a specific metric in the way of PromeQL, take "http_requests_total" as an example, the display effect is as shown in the figure:

This shows that the Promethues monitoring system has been running at this time, and collected the relevant Metrics index data!

Grafana visual monitoring system

Grafana is a powerful cross-platform open source measurement analysis and visualization tool, which can display the collected index data in a customized graphical interface, and is often used as a visualization for time series data and application analysis. Grafana supports multiple data sources, such as InfluxDB, OpenTSDB, ElasticSearch, and Prometheus.

When we installed and deployed Prometheus-Operator in Kubernetes earlier, Grafana has already been integrated and running. You can query the actual running Pod of Grafana through the relevant commands of Kubernetes, and expose its Web port to the public, as shown below:

# View service node information kubectl-n monitoring get svc# exposes promethues-operator-grafana outside the cluster using nodeport method. Specify port 30441 kubectl patch svc promethues-operator-grafana-n monitoring-p'{"spec": {"type": "NodePort", "ports": [{"port": 80, "targetPort": 3000, "nodePort": 30441}]}}'

It should be noted that since the default port for Grafana applications is 80, in order to avoid experimental environment conflicts, the target container port is specified as 3000, and the node port is finally mapped to 30441. When finished, the browser enters URL:

# IP address is the node http://10.211.55.11:30441 where the mapping command is executed

If the mapping is normal, the login interface of the Grafana visual graphical interface will be returned, as shown in the figure:

The default login password here is: admin/prom-operator. After entering, you can enter the main interface of Grafana, as shown below:

You can see that the deployed Grafana has built in many enterprise monitoring Dashboard for Kubernetes platform by default, such as "Kubernetes/API server" and "Kubernetes/Kubelet" for Kubernetes cluster components, "Kubernetes/Compute Resources/Pod" and "Kubernetes/Compute Resources/Workload" for Kubernetees computing resources, and so on.

Here we find a "Nodes" monitoring Dashboard for the Kubernetes physical node. Click to open and the monitoring effect is shown below:

The Dashboard shown in the figure above shows the use of hardware resources such as CPU, load, memory, disk I / O, disk space, network transfer, and so on, where the Kubernetes cluster resides. From these rich views, we can see Grafana's powerful visualization ability of monitoring indicators!

After reading the above, do you have any further understanding of the Kubernetes-based micro-service monitoring system? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.