Shulou(Shulou.com)06/02 Report--

First, introduce Prometheus

Prometheus (Prometheus) is an open source combination of monitoring-alarm-time series database, which was originally developed by SoundCloud. As it grows, more and more companies and organizations accept the adoption of Prometheus, and society is very active, so they turn it into an open source project and have companies to operate it. Google SRE also mentioned in his book that a similar implementation to their BorgMon monitoring system is Prometheus. Nowadays, the most common Kubernetes container management systems are usually monitored with Prometheus.

The basic principle of Prometheus is to capture the status of monitored components periodically through HTTP protocol. The advantage of this is that any component can access the monitoring system as long as it provides HTTP interface, without any SDK or other integration process. This is ideal for virtualized environments such as VM or Docker.

Prometheus should be one of the few monitoring systems suitable for Docker, Mesos and Kubernetes environments.

Compared with other monitoring systems, the main features of Prometheus are:

A multidimensional data model (the time series is defined by the indicator name and sets the key / value size).

Very efficient storage, an average of a sample data accounts for about 3.2 million of the time series, sampling every 30 seconds, holding for 60 days, consuming about 228G of disk.

A flexible query language.

Does not rely on distributed storage, a single server node.

Time collection is done through the PULL model on HTTP.

Push time is supported through intermediate gateways.

Discover the target through service discovery or static configuration.

Multiple modes of graphics and dashboard support.

II. Overview of Prometheus architecture

The figure illustrates the overall architecture of Prometheus and some of its ecosystem components:

Its service process is that Prometheus daemon is responsible for regularly fetching metrics (index) data on the target, and each fetching target needs to expose an interface of the http service to it.

Prometheus: support to specify crawling targets through configuration files, text files, zookeeper, Consul, DNS SRV lookup, etc. Support many ways of chart visualization, such as the very beautiful Grafana, its own Promdash, and its own template engine, etc., but also provides HTTP API query, custom output.

Alertmanager: is a component independent of Prometheus, which can support the query statement of Prometheus and provide a very flexible alarm mode.

PushGateway: this component supports Client to actively push metrics to PushGateway, while Prometheus only regularly goes to Gateway to grab data.

If there are users who have used statsd, they will feel that this is very similar, except that the statsd is sent directly to the server, while the Prometheus mainly depends on the process to grab it.

Most Prometheus components are written in Go, and they can be easily built and deployed as static binaries. Visit prometheus.io for complete documentation, examples, and guidelines.

3. Prometheus four data types Counter

Counter is used to accumulate values, such as the number of requests, tasks completed, and errors that have occurred. It keeps increasing, but it won't decrease. After the process is restarted, it will be reset.

For example, http_response_total {method= "GET", endpoint= "/ api/tracks"} 100 grabs http_response_total {method= "GET", endpoint= "/ api/tracks"} 100 after 10 seconds.

Gauge

Gauge general values, such as temperature changes, memory usage changes. Variable big, variable small. After the process is restarted, it will be reset.

For example: memory_usage_bytes {host= "master-01"} 100

< 抓取值、memory_usage_bytes{host="master-01″} 30、memory_usage_bytes{host="master-01″} 50、memory_usage_bytes{host="master-01″} 80 < 抓取值。 Histogram Histogram（直方图）可以理解为柱状图的意思，常用于跟踪事件发生的规模，例如：请求耗时、响应大小。它特别之处是可以对记录的内容进行分组，提供count和sum全部值的功能。 例如：{小于10=5次，小于20=1次，小于30=2次}，count=7次，sum=7次的求和值。 Summary Summary和Histogram十分相似，常用于跟踪事件发生的规模，例如：请求耗时、响应大小。同样提供 count 和 sum 全部值的功能。 例如：count=7次，sum=7次的值求值。 它提供一个quantiles的功能，可以按%比划分跟踪的结果。例如：quantile取值0.95，表示取采样值里面的95%数据。 五、实验环境docker01docker02docker03192.168.1.11192.168.1.13192.168.1.20NodeEXporterNodeEXporterNodeEXportercAdvisorcAdvisorcAdvisorPrometheus Server空空Grafana空空 全部关闭防火墙，禁用selinux 需要部署的组件： Prometheus Server:普罗米修斯的主服务器。 Prometheus是一个开源的服务监控系统，它通过HTTP协议从远程的机器收集数据并存储在本地的时序数据库上。 多维数据模型（时序列数据由metric名和一组key/value组成）在多维度上灵活的查询语言(PromQl)不依赖分布式存储，单主节点工作.通过基于HTTP的pull方式采集时序数据可以通过push gateway进行时序列数据推送(pushing)可以通过服务发现或者静态配置去获取要采集的目标服务器多种可视化图表及仪表盘支持Prometheus通过安装在远程机器上的exporter来收集监控数据，后面我们将使用到node_exporter收集系统数据。 [NodeEXporter]():负责收集Host硬件信息和操作系统信息。 [cAdvisor]():负责收集Host.上运行的容器信息。 Grafana:负责展示普罗米修斯监控界面。 Grafana 是一个开箱即用的可视化工具，具有功能齐全的度量仪表盘和图形编辑器，有灵活丰富的图形化选项，可以混合多种风格，支持多个数据源特点。 这些可以直接docker pull下载镜像（现在是本地导入镜像） 本地上传镜像 docker01 [09:05:42][docker01$ docker load -i node-exporter.tar && docker load -i mycadvisor.tar && docker load -i prometheus.tar && docker load -i grafana.tar docker02和docker03 [09:05:22]docker03]$ docker load -i node-exporter.tar && docker load -i mycadvisor.tar六、各主机部署1) 3个节点，全部部署node-EXporter,和cAdvisor.部署安装node-EXporter收集节点硬件和操作系统信息。[09:21:03[docker01]$ docker run -d -p 9100:9100 -v /proc:/host/proc -v /sys:/host/sys -v /:/rootfs --net=host prom/node-exporter --path.procfs /host/proc --path.sysfs /host/sys --collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"//部署node-EXporter,收集硬件和系统信息。 PS: 注意，这里使用了--net=host， 这样Prometheus Server可以直接与Node- EXporter通信。 验证:打开浏览器验证结果。http://192.168.1.11:9100/，http://192.168.1.13:9100/，http://192.168.1.20:9100/

Deploy and install cAdvisor to collect node container information. [09:39:10 [docker01] $docker run-v /: / rootfs:ro-v / var/run:/var/run/:rw-v / sys:/sys:ro-v / var/lib/docker:/var/lib/docker:ro-- detach=true-- name=cadvisor-- net=host google/cadvisor

Validation: open a browser to verify the results. Http://192.168.1.11:8080, http://192.168.1.13:8080, http://192.168.1.20:8080

2) deploy the Prometheus Server service on docker01.

Before deploying prometheus, we need to modify its configuration file, so let's run a container and copy its configuration file first.

09:51:22] [docker01] $docker run-d-p 9090 name prometheus-- net=host prom/prometheus// opens a Prometheus [09: 51: 00 [docker01] $docker cp prometheus:/etc/prometheus/prometheus.yml. / / copy the configuration file of Prometheus to modify the configuration file of Prometheus locally, and add a listening port (29 lines) [09:55:53] [docker01] [~] $vim prometheus.yml / / modify the configuration file. The monitoring item of prometheus is specified here. Including it will also monitor the data received by its own mobile phone. -targets: ['localhost:9090','localhost:8080','localhost:9100','192.168.1.13:8080','192.168.1.13:9100','192.168.1.20:8080' '192.168.1.20 prometheus// 9100'] rerun prometheus container [10:00:27] [docker01] [~] $docker rm-f prometheus// delete prometheus container [10:02:45] [docker01] [~] $docker run-d-p 9090 prometheus// 9090-- name prometheus-- net=host-v / root/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus// run a new prometheus container browser to access Authentication: http://192.168.1.11:9090/graph

Ps: you can see our various monitoring items here.

If you suspend a virtual machine now (continue to run after testing)

3) in docker01. Deploy the grafana service to display the data collected by prometheus. [root@docker01 ~] # mkdir grafana-storage// create a directory to collect information [root@docker01 ~] # chmod 777 grafana-storage/// gives 777 permissions [root@docker01 ~] # docker run-d-p 3000-name grafana- v / root/grafana-storage:/var/lib/grafana-e "GF_SECURITY_ADMIN_PASSWORD=123.com" grafana/grafana browser access verification:

(user name: admin, password: 123.com)

Add data source

PS: see this prompt, indicating that the prometheus and grafana services are connected properly.

At this point, although grafana has collected the data, how to display it is still a problem. Grafana supports custom display information, but it is very troublesome to customize it, but fortunately, grafana officially provides us with some templates for us to use.

Grafana official website: https://grafana.com/docs/grafana/latest/

Select a template, and then we have two ways to apply it. The first way: use the template through the JSON file.

After the download is complete, go to the grafana console

The second way to import templates: * *

You can use the ID number of the template directly.

/ / this id is not easy to use and changed to 8321.

After copying the template id, go to the grafana console

Eliminate the wrong train of thought

Whether the firewall is off and whether selinux is disabled

Whether the host name has been changed

Whether the mirror image is normal

Whether the directory is mounted correctly when each service is started

Grafana service, whether to create the required directory and whether the directory has permissions

Whether the Prometheus service modifies the configuration file

Summary

Congratulations! You have set up Docker that can be used by the Prometheus server, Node Exporter, Grafana, and so on. Although these are currently running on the same machine, they are for demonstration purposes only. In production settings, the node exporter, multiple Prometheus servers (depending on the needs of the organization), and a single Grafana server are typically run on each monitored computer to draw data from those servers.

Reference: https://www.cnblogs.com/xiao987334176/p/9930517.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.