Shulou(Shulou.com)06/02 Report--

Blog outline:

1. Monitoring of Docker

II. Brief introduction of prometheus

III. Composition and structure of Prometheus

IV. Deploy prometheus

1) Environmental preparation

2) deploy prometheus

3) configure Peometheus monitoring to realize alarm

I. Monitoring of Docker (1) the monitoring command included with docker [root@localhost ~] # docker top wordpress_wordprss_1 / / View the usage status of the container UID PID PPID C STIME TTY TIME CMDroot 5601 5569 0 20:53? 00:00:00 apache2-DFOREGROUND33 6073 5601 0 20:54? 00:00:00 apache2-DFOREGROUND33 6074 5601 0 20:54? 00:00:00 apache2-DFOREGROUND33 6075 5601 0 20:54? 00:00:00 apache2-DFOREGROUND33 6076 5601 0 20:54? 00:00:00 apache2-DFOREGROUND33 6077 5601 0 20:54 ? 00:00:00 apache2-DFOREGROUND33 6096 5601 0 20:54? 00:00:00 apache2-DFOREGROUND33 6098 5601 0 20:54? 00:00:00 apache2-DFOREGROUND33 6099 5601 0 20:54? 00:00:00 apache2-DFOREGROUND33 6100 5601 0 20:54? 00:00:00 apache2-DFOREGROUND33 6155 5601 0 20:57? 00:00:00 Apache2-DFOREGROUND [root@localhost ~] # docker stats wordpress_wordprss_1 [root@localhost ~] # docker logs wordpress_wordprss_1 / / these are all monitoring commands that come with the container itself (2) sysdig [root@localhost ~] # docker run-it-- rm-- name sysdig-- privileged=true-- volume=/var/run/docker.sock:/host/var/run/docker.sock-- volume=/dev:/host/dev-- volume=/proc:/host/proc:ro-- Volume=/boot:/host/boot:ro-- volume=/lib/modules:/host/lib/modules:ro-- volume=/usr:/host/usr:ro sysdig/sysdig// creates a container and automatically enters the container / /-- rm: deleted as you exit the container / /-- privileged=true: give special permissions; root@711dbeb59fdd:/# csysdig / / execute this command

As shown in the figure:

In the picture, you can use keyboard and mouse to operate!

(3) scope [root@localhost ~] # curl-L git.io/scope-o / usr/local/bin/scope [root@localhost ~] # chmod + x / usr/local/bin/scope / / download the installation script [root@localhost ~] # scope launch / / launch in container mode. Weave Scope is listening at the following URL (s): * http://172.21.0.1:4040/ * http://192.168.122.1:4040/ * http://172.22.0.1:4040/ * http://172.20.0.1:4040/ * http://172.18.0.1:4040/ * http://172.19.0.1:4040/ * http://192.168.1. 1DUR 4040Compact / access based on the prompt at the end

As shown in the figure:

According to the tips in the picture, you can click to test!

If you want to monitor the two servers: the hostnames must be distinguished as follows:

[root@dockerA ~] # curl-L git.io/scope-o / usr/local/bin/scope [root@dockerA ~] # chmod + x / usr/local/bin/scope [root@dockerA ~] # scope launch 192.168.1.1 192.168.1.2 / / preferred to specify local IP Then specify the other party's IP [root@dockerA ~] # docker run-itd-- name http httpd / / run a container to test [root@dockerB ~] # curl-L git.io/scope-o / usr/local/bin/scope [root@dockerB ~] # chmod + x / usr/local/bin/scope [root@dockerB ~] # scope launch 192.168.1.2 192.168.1.1 [root@dockerB] # docker run-itd-- name nginx nginx

Visit (either dockerA or dockerB) for testing:

II. Brief introduction of prometheus

Prometheus is an open source system monitoring and alarm framework. It collects metrics from configured targets at a given interval, evaluates rule expressions, displays results, and triggers alerts when certain conditions are found to be true.

As a new generation of monitoring framework, Prometheus has the following characteristics:

Powerful multi-dimensional data model:

(1) time series data are distinguished by metric name and key-value pair.

(2) all metrics can set arbitrary multi-dimensional tags.

(3) the data model is more casual and does not need to be deliberately set to a dot-delimited string.

(4) data models can be aggregated, cut and sliced.

(5) double-precision floating-point type is supported, and the label can be set to full unicode (unified code); flexible and powerful query statements: multiple metrics can be multiplied, added, joined, and fractional bits can be taken in the same query statement; easy to manage: do not rely on distributed storage; use pull mode to collect time series data; use push gateway to push time series data to Prometheus server. Targets;, which can be monitored through service discovery or static configuration, has a variety of visual graphical interfaces; it is easy to scale. Composition and structure of Prometheus

Prometheus contains a number of components, many of which are optional, commonly used components are:

Prometheus Server: used to collect and store time series data; Client Library: client library that generates the corresponding metrics for the services to be monitored and exposes it to Prometheus server;Push Gateway: mainly for short-term jobs. Because this kind of jobs exists for a short time, it may disappear before Prometheus comes to pull. To this end, the jobs can push their metrics;Exporters directly to the Prometheus server side: the metrics used to expose the existing third-party services to the Prometheus;Alertmanager: after receiving the alerts from the Prometheus server side, the data will be deduplicated, grouped, and routed to the acceptance mode of the peer, and an alarm will be issued.

. Wait, there are many more, here are a few commonly used components!

The architecture diagram in the official Prometheus documentation:

In the official architecture diagram, the main module blocks include: Prometheus server, exporters, Pushgateway, PromQL, Alertmanager and graphical interface

The general workflow is as follows:

(1) Prometheus server regularly pulls metrics from configured jobs or exporters, or receives metrics from Pushgateway, or pulls metrics from other Prometheus server

(2) Prometheus server stores the collected metrics locally and runs the defined alert.rules to record new time series or push alerts to Alertmanager

(3) Alertmanager processes the received alarms and issues alarms according to the configuration file

(4) visually collect data in the graphical interface.

IV. Deployment of prometheus1) Environment preparation

Note: the most basic docker environment must be available on the above three servers. The docker version of the above environment is 18.09.0!

The components required for the above environment are as follows:

Prometheus server: Prometheus master server (port: 9090); NodeEXporter: responsible for collecting Host hardware information and operating system information (port: 9100); cAdvisor: responsible for collecting container information running on Host (port: 8080); Grafana: responsible for displaying Prometheus monitoring interface (3000) Alertmanager: used to receive the alarm message sent by Prometheus, and execute the set alarm mode and alarm content (also deployed on the dockerA host, port: 9093)

The relationship of each component: NodeEXporter and cAdvisor are responsible for collecting information and sending it to Prometheus server, and then Prometheus server gives it to Grafana for graphical display. If you need to give an alarm, prometheus will send a message to the Alertmanager component!

Experimental environment, for the sake of simplicity, turn off the firewall, SELinux, the actual environment needs to open the corresponding port!

2) deploy prometheus (1) run node-exporter container

NodeEXporter is mainly responsible for collecting Host hardware information and operating system information!

[root@dockerA ~] # docker run-d-- name node- p 9100lu 9100-v / proc:/host/proc-v / sys:/host/sys-v /: / rootfs-- net=host prom/node-exporter-- path.procfs / host/proc-- path.sysfs / host/sys-- collector.filesystem.ignored-mount-points "^ / (sys | proc | dev | host | etc) ($| /)" / create a container called node using prom/node-exporter image to collect hardware and system information / /-- net=host indicates that Prometheus server can communicate directly with node-exporter; / / and maps port 9100

After the execution is completed, the client accesses it using a browser, as shown in the figure:

Visiting the above page indicates that the node-exporter component has been installed successfully!

Since this NodeEXporter component needs to be on three docker host hosts, the above command needs to be executed on both other hosts. After the execution is complete, use your own browser to access the test!

(2) run cAdvisor container

CAdvisor is mainly responsible for collecting container information running on Host!

[root@dockerA] # docker run-v /: / rootfs:ro-v / var/run:/var/run/:rw-v / sys:/sys:ro-v / var/lib/docker:/var/lib/docker:ro-p 8080pur808080-- detach=true-- name=cadvisor-- net=host google/cadvisor

Client access test:

Visiting the above page indicates that the cAdvisor component has been installed successfully!

Similarly, this cAdvisor component also needs to be installed on all three docker host! Therefore, the above commands also need to be executed on the other two hosts, and after the execution is completed, test by yourself!

(3) run the Prometheus server container (just execute it on the dockerA host! )

Prometheus is Prometheus's main server!

Before deploying Prometheus, you need to make changes to its configuration file, so first run a Prometheus container to copy its configuration file locally to make it easier to modify.

[root@dockerA ~] # docker run-d-p 9090name prometheus-- net=host prom/prometheus// runs a Prometheus container to get its configuration file to the local [root@dockerA ~] # docker cp prometheus:/etc/prometheus/prometheus.yml. / / copy the main configuration file in the Prometheus container to the local [root@dockerA ~] # vim prometheus.yml / / Edit the main configuration file-targets: ['localhost:9090','localhost:8080' 'localhost:9100','192.168.1.2:8080','192.168.1.2:9100','192.168.1.3:8080','192.168.1.3:9100'] / / this item originally existed Just modify it! / / specify ports 9090, 8080 and 9100 to monitor the local machine, and add ports 8080 and 9100 to the other two docker hosts. / / Port 8080 is running a cAdvisor service / / Port 9100 is running a node-exporter service / / Port 9090 is running a Prometheus service [root@dockerA ~] # docker rm prometheus-f / / delete the container you just ran prometheus [root@dockerA ~] # docker run-d-p 9090Prometheus-- name prometheus-- net=host-v / root/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus// run a prometheus container again Mount the configuration file you just modified into the container

Client access test:

(4) run the grafana container (also just run it on the dockerA host! )

Grafana is mainly responsible for displaying Prometheus monitoring interface and providing us with a good graphical interface!

[root@dockerA ~] # mkdir grafana-storage [root@dockerA ~] # chmod 777-R grafana-storage// creates a directory with the permission of [root@dockerA ~] # docker run-d-p 3000 chmod 3000-- name grafana- v / root/grafana-storage:/var/lib/grafana-e "GF_SECURITY_ADMIN_PASSWORD=123.com" grafana/grafana// "- e" option means to modify the environment variable inside the container and change the password of the admin user to 123.com

Client access test:

After the above configuration is completed, we need to configure it in what form to show us that it can be customized, but it is very troublesome. I choose to go directly to the grafana official website to find a ready-made template. As shown in the figure:

Import the official grafana template into the web page provided by our grafana container in two ways:

1) the first way:

Select your favorite template on the grafana official website, and click to enter, as shown in the figure:

After downloading, go back to the web page provided by the grafana container you built, as shown below:

But if you look closely, you will find that some information about this template can not be detected, so here is to show a way to import the template. Personal recommendation to use the second way!

2) the second way:

After selecting the appropriate template, record its ID number, as shown in the figure:

After recording the ID number of the template, you can also return to the web page provided by the grafana container you built, as shown below:

At this point, the monitoring of the web interface is complete!

3) configure Peometheus monitoring to realize alarm

There are several ways of Prometheus alarm, such as: mailbox, Wechat, nails and so on. This case uses mailbox alarm.

(1) run the alertmanager container

The Alertmanager component is mainly used to receive the alarm information sent by Prometheus, and to execute the set alarm mode and alarm content.

Just deploy on the dockerA host! The methods are as follows:

[root@dockerA] # docker run-d-- name alertmanager-p 9093 name alertmanager 9093 prom/alertmanager// run a random container Its purpose is to get the configuration file of the service in the container to the local [root@dockerA ~] # docker cp alertmanager:/etc/alertmanager/alertmanager.yml. / / the yml configuration file of the altermanager service to the local [root@dockerA ~] # vim alertmanager.yml / / edit the configuration file global: resolve_timeout: 5m smtp_from: '1454295320roomq.com' # sender information smtp_smarthost:' Smtp.qq.com:465' smtp_auth_username: '1454295320roomqq.com' # recipient information smtp_auth_password: 'gfuxsudyqyulbaad' # uses the authorization code generated by QQ Mail smtp_require_tls: false smtp_hello:' qq.com'route: group_by: ['alertname'] # uses the default group group_wait: 5s group _ interval: 5s repeat_interval: 5m receiver: 'email' receivers:- name:' email' email_configs:-to: '1454295320roomq.com' / / sender information send_resolved: true / / when the container returns to normal An email will also be sent: inhibit_rules:-source_match: severity: 'critical' target_match: severity:' warning' equal: ['alertname',' dev', 'instance'] / / comments have been made on common changes in the configuration file.

First-level fields in the alertmanager.yml configuration file:

Global: global configuration (including timeout for alarm resolution, SMTP-related configuration, API addresses and other new messages notified by various channels); route: set alarm distribution policy; receivers: configure alarm message recipient information; inhibit_rules: suppression rule configuration, when there are alerts matching another set, suppression rules will only be used for one set of matching [root@dockerA ~] # docker rm-f alertmanager// delete the original alertmanager container [root@dockerA ~] # docker run-d-- name alertmanager-p 9093-v / root/alertmanager.yml:/etc/alertmanager/alertmanager.yml prom/alertmanager// rerun the alertmanager container and mount the configuration file to the container / / after it is recommended to run the container, use docker ps | grep alertmanager to ensure the proper operation of the container / / if the configuration file is written incorrectly Then this container cannot be started. (2) configure alertmanager alarm rules [root@dockerA] # mkdir-p prometheus/rules & & cd prometheus/rules// create a directory for storing rules [root@dockerA rules] # vim node-up.rules / / write rules groups:- name: node-up / / Custom name rules:-alert: node-up expr: up {job= "prometheus" } = = 0 / / the name of job must correspond to-job_name: 'prometheus' in the prometheus configuration file. For: 15s labels: severity: 1 team: node annotations: summary: "{{$labels.instance}} has stopped running for more than 15s!"

You can modify it according to the above configuration file. If you want to write the alarm rules yourself, you can refer to its official documentation, as shown in the figure:

(3) Associate Prometheus [root@dockerA ~] # vim prometheus.yml 8 alerting: 9 alertmanagers: 10-static_configs: 11-targets: 12-192.168.1.1 vim prometheus.yml 9093 / / this line changes the original content to IP+ port 13 14 # Load rules once and periodically evaluate them according to the global 'evaluat ion_interval'. of the alertmanager container 15 rule_files: 16-"/ usr/local/prometheus/rules/*.rules" / / this line needs to be added manually. Specify the path within the container [root@dockerA ~] # docker rm-f prometheus/ / delete the container after modifying the configuration to a file. Rerun a new container [root@dockerA] # docker run-d-p 9090 name prometheus-- net=host-v / root/prometheus.yml:/etc/prometheus/prometheus.yml-v / root/prometheus/rules/node-up.rules:/usr/local/prometheus/rules/node-up.rules prom/prometheus// specify the path to the rule file written in step (2)

To prevent possible format errors, attach a screenshot as follows:

At this point, if there is an exception in the target in the prometheus page (such as an outage), an alarm will be sent to your mailbox.

I manually stopped the container and received the following email:

In order to pursue beauty, we take to set the alarm message template!

(4) set alarm message template [root@dockerA ~] # cd prometheus [root@dockerA prometheus] # mkdir alertmanager-tmpl [root@dockerA prometheus] # cd alertmanager-tmpl/ [root@dockerA prometheus] # vim email.tmpl {{define "email.from"}} 1454295320@qq.com {{end} {{define "email.to"}} 1454295320@qq.com {{end} {{define "email.to.html"}} {{range .Alerts}} = start=

Alarm program: prometheus_alert

Alarm level: {{.Labels.alarm}} level

Alarm type: {{.Labels.alertname}}

Failed host: {{.Labels.instance}}

Alarm topic: {{.Annotations.summary}}

Trigger time: {{.StartsAt.Format "2019-08-04 16:58:15"}}

= end=

{{end}} {{end}} [root@dockerA ~] # vim alertmanager.yml global: resolve_timeout: 5m smtp_from: '1454295320roomq.com' smtp_smarthost: 'smtp.qq.com:465' smtp_auth_username:' 1454295320Secretq.com 'smtp_auth_password:' flnuwdktcbzwffag' smtp_require_tls: false smtp_hello: 'qq.com'templates: / / add this line -'/ etc/alertmanager-tmpl/*.tmpl' / / specify the path of the template in the container route: group_by: ['alertname'] group_wait: 5s group_interval: 5s repeat_interval: 5m receiver:' email'receivers:- name: 'email' email_configs:-to:' {template "email.to"}}'/ / must correspond to the template Html:'{template "email.to.html". }'/ / must match the corresponding send_resolved: trueinhibit_rules:-source_match: severity: 'critical' target_match: severity:' warning' equal: ['alertname',' dev''in the template 'instance'] [root@dockerA ~] # docker rm alertmanager- fplink / delete container [root@dockerA ~] # docker run-d-- name alertmanager- p 9093purl 9093-v / root/alertmanager.yml:/etc/alertmanager/alertmanager.yml-v / root/prometheus/alertmanager-tmpl:/etc/alertmanager-tmpl prom/alertmanager// create the template file created locally by the container / / after the creation is completed Make sure the container is running properly

At this point, a new alarm template is also generated. If the following container has Down, a new email will be sent to you. When it returns to normal, it will also be sent. Similarly, the content of the email is in the wrong format, but you can receive the alarm message normally. If you want to change the alarm template, you can refer to the official documentation.

The alarm message received by the mailbox is as follows:

The effect is much better than the original! Just briefly introduce this one. If you are interested, you can refer to the documentation on the official website and write it on your own!

-this is the end of this article. Thank you for reading-

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.