Shulou(Shulou.com)05/31 Report--

How to use the monitoring weapon Prometheus, in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Foreword:

Kubernetes, as the most popular container management platform, not only brings convenience to application deployment, operation and maintenance, but also brings new challenges to application and performance monitoring. Let's share with you a very hot open source monitoring tool Prometheus. Let's take a look at how it takes into account traditional application monitoring, host performance monitoring and

A brief introduction to Prometheus

What is Prometheus?

Prometheus is an open source system monitoring and alarm tool, originally built in SoundCloud. Since the launch of Prometheus in 2012, many companies have used it to build monitoring and alarm systems. At the same time, the project has a very active developer and user community.

It is now an open source project independent of any company, and in order to emphasize this point and clarify the management structure of the project, Prometheus joined the CNCF Foundation in 2016 as the second managed project after Kubernetes.

What are the characteristics of Prometheus?

A multidimensional data model (based on the KBV key-value pair of time series).

Do not rely on distributed storage, node autonomy.

Pull mode based on HTTP collects time series data.

You can use pushgateway (optional middleware for prometheus) to implement the push pattern.

You can use dynamic service discovery or statically configure the collected target machine.

Support a variety of graphics and dashboards.

Prometheus is suitable for scenarios.

Before choosing Prometheus as a monitoring tool, you should be clear about its scope of application and the scenarios in which it does not apply.

Prometheus is very good at recording pure-value time series. It is not only suitable for server-centric monitoring, but also for highly dynamic service-oriented architecture monitoring.

In the monitoring of micro-services, Prometheus's support for multi-dimensional data collection and query is also a special advantage.

Prometheus puts more emphasis on reliability and can view the statistics of the system even in the case of failure. Weigh the pros and cons to ensure the availability of the entire system at the expense of possible loss of a small amount of data. Therefore, it is not suitable for systems that require 100% data accuracy, such as real-time billing systems (involving money).

II. Prometheus architecture diagram

The above picture is the architecture diagram of Prometheus, from which we can see the architecture design concept of Prometheus, centralized data collection and analysis.

2. Pushgateway: in response to the plug-ins provided in some push scenarios, the monitoring data is first pushed to the pushgateway, and then the pull is collected by the server. (if the data on the pushgateway does not change during the server collection interval, the server will collect the same data twice, only with a different timestamp)

3. Prometheus targets: the probe (exporter) provides the acquisition interface, or the acquisition interface provided by the application itself supports the Prometheus data model.

4. Service discovery: support service discovery based on the way the configuration file_sd monitors the local configuration file (you need to modify the local configuration file with other tools). At the same time, you can configure the API listening on the Kubernetes to dynamically discover the service.

5. Alertmanager:Prometheus alarm plugin, which supports sending alarm to email, Pagerduty,HipChat, etc.

III. Detailed explanation of Prometheus architecture

Next, let's take a look at how the various components in the rometheus architecture work together to accomplish monitoring tasks.

Prometheus server and targets

The monitoring of all commonly used middleware or third-party tools can be completed by using the probes provided by Prometheus officials or third parties.

I mentioned earlier that Prometheus is a centralized data acquisition and analysis, so what does the probe (exporter) do here?

In the above figure, the hardware and system monitoring probe node exporter obtains the memory information of the machine through the getMemInfo () method, and then corresponds the total memory data of the machine to the index node_memory_MemTotal.

Jenkins probe Jenkins Exporter obtains the number of job of Jenkins by visiting the api of Jenkins and corresponds to the index Jenkins_job_count_value.

The function of the probe is to collect the monitoring data by calling the application or system interface and return the corresponding index to prometheus server. (the probe does not have to be deployed on the same machine as the monitored application)

Generally speaking, the Prometheus data acquisition process is to configure the port address exposed by the probe and the collection interval in the Prometheus server, and the Prometheus accesses the probe through http according to the configured time interval. At this time, the probe obtains the monitoring data by calling the interface and returns the corresponding index to Prometheus server for storage. (if the probe does not finish collecting data within the acquisition interval configured by Prometheus, this part of the data will be lost.)

Prometheus alerting

How does Prometheus serve complete the alarm according to the collected monitoring data and alertmanager?

An example of a common alarm is to send an alarm when the available memory of the host is less than 20% of the total memory. According to the host performance index collected by Prometheus server, we can configure such a rule node_memory_Active/node_memory_MemTotal < 0.2 minute Prometheus server to analyze the collected data. When this condition is met, alarm information is sent to alertmanager,alertmanager to process alarm information according to local configuration and sent to third-party tools to be received by the relevant responsible person.

Here, Prometheus server is mainly responsible for analyzing data according to alarm rules and sending alarm information to alertmanager,alertmanager, while processing alarm information and sending alarm information according to configuration.

Grouping: alarms with the same monitoring targets are grouped. In the event of a power outage, you should receive a single message containing all the affected machines that are down, rather than sending an alarm message for each machine that is down.

Suppression: suppression refers to the mechanism that stops sending other alarms caused by the alarm when the alarm is issued. If the machine network is unreachable, other alarms caused by network problems will no longer be sent.

Silence: the alarm information is filtered according to the defined rules, and the matching alarm information is not sent.

Service discovery

Prometheus supports a variety of ways of service discovery, and here we mainly introduce the way of file_sd mentioned in the architecture diagram. As mentioned earlier, the data collection configuration of Prometheus server is all through the configuration file, so what should be done for service discovery?

File_sd_configs is used here to specify the file that defines the collection target. Prometheus server will dynamically detect the changes in the configuration file to update the collection target information. Now as long as you can update this configuration file, you can dynamically modify the configuration of the collection target.

The consul+consul template approach is used here. Update KLV in consul when adding or decreasing probes (increasing or decreasing acquisition targets). For example, add a new probe, add the following record Prometheus/linux/node/10.15.15.132=10.15.15.132:9100, and then configure consul template to monitor the changes of KLV in the Prometheus/linux/node/ directory of consul, and dynamically generate Prometheus server configuration file discovery.yml according to the value of KLV and the discovery.ctmpl template defined in advance.

Web UI

At this point, the data collection and alarm configuration have been completed, and it is time to show a wave of results through the page.

Grafana has done a good job of supporting Prometheus, adding Prometheus data sources to Grafana, and then we can use PromQL queries combined with grafana's powerful graphical capabilities to configure our performance monitoring page.

Federal mode

Centralized data collection, storage, analysis, and does not support cluster mode. The performance problems are obvious. Prometheus offers a federated deployment in which Prometheus server can collect data from other Prometheus server.

One might ask, won't all the final data be aggregated to the global node of Prometheus?

This is not the case, we can complete the analysis and processing in the shard node, and then the global node directly collects and analyzes the processed data for display.

For example, the result of defining the percentage of available memory in the shard node job:memory_available:proportion is (node_memory_MemFree + node_memory_Buffers + node_memory_Cached) / node_memory_MemTotal, so that the aggregation operation can be completed on the shard node, and then the global node can directly collect and process the data instead of collecting scattered indicators such as node_memory_MemFree.

4. Prometheus monitors Kubernetes

Heapster+influxdb,heapster officially recommended a performance monitoring solution. According to the defined intervals, Kubernetes obtains the performance data about pod and container from Advisor and stores them in the time series database influxdb.

You can also use grafana to configure influxdb's data source and configure dashboard for presentation. And the automatic scaling feature (Horizontal Pod Autoscaling) of pod in Kubernetes is also based on heapster, which supports dynamic scaling according to cpu metrics by default, or you can customize the expansion to use other metrics.

Now, Heapster as an open source monitoring solution under Kubernetes has been abandoned by it (https://github.com/kubernetes/heapster), and Prometheus has become the monitoring solution officially recommended by Kubernetes.

Prometheus also obtains the performance monitoring data of pod and container through Kubernetes's cAdvisor interface (/ api/v1/nodes/$ {1} / proxy/metrics/cadvisor). At the same time, it can use Kubernetes's Kube-state-metrics plug-in to obtain the status of various resource objects on the cluster, such as Pod, DaemonSet, Deployment, Job, CronJob, and so on, which reflects the status of applications using these resources.

At the same time, the information of node,service,pod,endpoints,ingress and other services is obtained through Kubernetes api, and then the collection target is obtained by matching the values in the annotations.

It is mentioned above that Prometheus can realize service discovery through the api interface of Kubernetes, and configure pod,service that matches and defines annotation parameters as the collection target. Then the problem to be solved now is the problem of probe to application deployment configuration.

Here we use the sidecar mode of Kubernetes's pod deployment. A single application pod deploys two containers. Taking advantage of the isolation feature of namespace that only shares the network in a single pod, the probe runs with the application, and the localhost can be used to access the application port directly. In the pod comments, only the port of the probe (prometheus.io/port: "9104") can be exposed.

Prometheus server annotates the pod of prometheus.io/scrape: "true" according to the configuration matching definition, and splices the pod ip with the port (prometheus.io/port: "9104") and path (prometheus.io/path: "/ metrics") defined in the note into the collection target http://10.244.3.123:9104/metrics. In this way, we can dynamically add applications that need to be collected.

This is the answer to the question about how to use the monitoring weapon Prometheus. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.