Shulou(Shulou.com)06/03 Report--

Prometheus (Prometheus) monitoring environment preparation:

Three docker hosts (centos7):

Docker01:172.16.1.30

Deployment Services: Prometheus server,Grafana,Node Exporter,cAdvrisor

Docker02:172.16.1.31

Deployment Services: Node Exporter,cAdvrisor

Docker03:172.16.1.32

Deployment Services: Node Exporter,cAdvrisor

Note: because it is a test environment, temporarily turn off the firewall and disable selinux. If it is a server published on a public network, you cannot turn off the firewall. You can set firewalld and iptables rules.

Prometheus server: Prometheus main server, listening port: 9090Node Exporter: collecting Host hardware and operating system information, listening port: 9100vAdvrisor: responsible for collecting container information running on Host, listening port: 8080Grafana: used to show Prometheus monitoring interface. Project actions:

1. Deploy Node Export (collect dockerhost information)

# all three dockerhost need to be deployed sequentially:

(1) download prom/node-exporter image: [root@sqm-docker02 ~] # docker pull prom/node-exporter (2) run Node Exporter container (mount system directory) [root@sqm-docker02 ~] # docker run-d-p 9100pur9100-v / proc:/host/proc-v / sys:/host/sys-v /: / rootfs-- net=host prom/node-exporter-- path.procfs / host/proc-- path.sysfs / host/sys-- collector .filesystem.committed-mount-points "^ / (sys | proc | dev | host | etc) ($| /)"

Parameter-- net=host: so that Prometheus server can communicate directly with Node Exporter

# check whether the container is running properly:

After all three are deployed, log in to the web page for verification:

Enter URL (for all three hosts): http://172.16.1.30:9100/

When you come in, you can see that the information is full of code, which is very unfriendly because we haven't deployed Grafana yet.

2. Deploy cAdvisor (collect container information)

# deploy on three dockerhost in turn

(1) download the google/cadvisor image [root@sqm-docker01 ~] # docker pull google/cadvisor (2) run the cAdvisor image: [root@sqm-docker01 ~] # docker run-v /: / rootfs:ro-v / var/run:/var/run:rw-v / sys:/sys:ro-v / var/lib/docker:/var/lib/docker:ro-p 8080 detach=true-- name=cadvisor-- net=host google/cadvisor

/ / check whether the container is running properly:

Log in to the web page to verify:

URL: http://172.16.1.30:8080

All kinds of information of dockerhost can be detected.

3. Deploy Prometheus server: (main service of Prometheus)

Deployed only on docker01:

(1) download Prometheus image: [root@sqm-docker01 ~] # docker pull prom/prometheus (2) get the configuration file of Prometheus First run a Prometheus-based image: [root@sqm-docker01 ~] # docker run-d-p 9090 net=host prom/prometheus 9090-- name prometheus-- net=host prom/prometheus (3) copy the configuration file in the prometheus container to the current directory: [root@sqm-docker01 ~] # docker cp prometheus:/etc/prometheus/prometheus.yml / root/ modify the .yml file: [root@sqm-docker01 ~] # vim prometheus.yml

Add the monitoring address of the service on this machine and the other two docker hosts: Port

(4) Delete the Prometheus container you just created (for copying configuration files only)

[root@sqm-docker01 ~] # docker stop prometheus [root@sqm-docker01 ~] # docker rm prometheus (5) run a Prometheus container: [root@sqm-docker01 ~] # docker run-d-p 9090V / root/prometheus.yml:/etc/prometheus/prometheus.yml-- name prometheus-- net=host prom/prometheus

/ / check whether the container is running properly:

Log in to the Prometheus web page:

URL: http://172.16.1.30:9090

Make sure that you monitor the host and port that just added three dockerhost to the configuration file, and that the status is up.

4. Deploy Grafana: (provide a friendly graphical interface)

Deploy on Prometheus server:

(1) download Grafana image: [root@sqm-docker01 ~] # docker pull grafana/grafana

(2) run Grafana: (set login password to 123.com) [root@sqm-docker01 ~] # mkdir grafana-storage [root@sqm-docker01 ~] # chmod 777-R grafana-storage/ [root@sqm-docker01 ~] # docker run-d-p 3000-name grafana- v / root/grafana-storage:/var/lib/grafana-e "GF_SECURITY_ADMIN_PASSWORD=123.com" grafana/grafana

If the above error occurs, you need to reload the process and restart the docker service.

Make sure the container is running:

Access the web interface (graphical):

URL: http://172.16.1.30:3000

Default user name: admin password: 123.com

The login interface is as follows:

(3) next, we deploy on the graphical web interface:

Click the icon named "add data source" on the home page-> Select "Prometheus" and click to enter the following interface

After the setting is successful, go to Grafana's official website: https://grafana.com/

Download the monitoring template

After the download is successful, go back to the graphical interface managed by Grafana.

Click to import the josn file you just downloaded:

、

After the import is successful, you can see the following monitoring interface:

Of course, you can also import other templates, you can download other templates on the official website just now.

3. Realize Prometheus mailbox alarm:

Must have been deployed in Prometheus server.

(1) download the Alertmanager alarm image [root@sqm-docker01 ~] # docker pull prom/alertmanager

(2) run alertmanager: [root@sqm-docker01 ~] # docker run-- name alertmanager-d-p 9093 prom/alertmanager

Test whether you can access the web interface:

URL: http://172.16.1.30:9093/

(3) copy its configuration file:

If you don't know its configuration file path, you need to go to the container to view it:

[root@sqm-docker01 ~] # docker cp alertmanager:/etc/alertmanager/alertmanager.yml / root/

Delete the container you just created: (just to copy the configuration file)

[root@sqm-docker01 ~] # docker stop alertmanager [root@sqm-docker01 ~] # docker rm alertmanager modify the .yml configuration file: [root@sqm-docker01 ~] # vim alertmanager.yml

Set the mailbox information to realize the alarm:

Note: for the smtp email protocol defined above, QQ Mail is 465.If you enter a different mailbox, you need to enter the corresponding smtp port number.

Configuration item explanation: global: global configuration. Alarm strategy, alarm channel and so on. Route: distribution policy. Receivers: the recipient. Inhibit_rules: suppression strategy. When alerts that match another set are stored, suppression rules prohibit alerts that match one set. (4) rerun the container: [root@sqm-docker01] # docker run-d-- name alertmanager-p 9093-v / root/alertmanager.yml:/etc/alertmanager/alertmanager.yml prom/alertmanager

/ / ensure the normal operation of the container:

(5) set Alertmanager alarm rules: [root@sqm-docker01 ~] # mkdir-p prometheus/rules [root@sqm-docker01 ~] # cd prometheus/rules/ write rule file: [root@sqm-docker01 rules] # vim node-up.rules

The above is customized and written by yourself. Of course, you can also log on to the official website of Prometheus and search the rule template to write:

/ / then modify the Prometheus configuration file: [root@sqm-docker01 ~] # vim prometheus.yml

Delete the Prometheus container that you just ran: [root@sqm-docker01 ~] # docker stop prometheus prometheus [root@sqm-docker01 ~] # docker rm prometheus prometheus// needs to rerun the container and mount the rule directory to the container: [root@sqm-docker01 ~] # docker run-d-p 9090V / root/prometheus.yml:/etc/prometheus/prometheus.yml-v / root/prometheus/rules:/usr/local/prometheus/rules-- name prometheus-- net=host prom/prometheus

After ensuring that the container is running properly, log in to the web page to view:

Make sure you can see the rules you just wrote.

(6) Test mailbox alarm:

Next, simulate a service outage on one of the hosts to verify whether QQ Mail can receive the email:

[root@sqm-docker01 ~] # docker stop cadvisor cadvisor

Log in to the Prometheus interface to see the service down:

When the service is down, the mailbox receives the alarm email with the prompt of "FIRING"

If your service resumes normal operation and the alarm rule is still triggered, an email will be sent to you informing you that the container has repaired "RESOLVED" (as shown below)

(7) set up alarm information template:

[root@sqm-docker01 ~] # mkdir alertmanager-tmpl [root@sqm-docker01 ~] # cd alertmanager-tmpl/ [root@sqm-docker01 alertmanager-tmpl] # vim email.tmpl

Write template script: (fill in QQ Mail who called the police)

{{define "email.from"}} 1234567890@qq.com {{end}}

{{define "email.to"}} 1234567890@qq.com {{end}}

{{define "email.to.html"}}

{{range .Alerts}}

= start=

Alarm program: prometheus_alert

Alarm level: {{.Labels.alarm}} level

Alarm type: {{.Labels.alertname}}

Failed host: {{.Labels.instance}}

Alarm topic: {{.Annotations.summary}}

Trigger time: {{.StartsAt.Format "2019-08-04 16:58:15"}}

= end=

{{end}}

{{end}}

/ / modify alertmanager configuration file: [root@sqm-docker01 ~] # vim alertmanager.yml

# Note indentation:

/ / Delete the original container: [root@sqm-docker01 ~] # docker stop alertmanager [root@sqm-docker01 ~] # docker rm alertmanager// rerun the container (remount the file) [root@sqm-docker01 ~] # docker run-d-- name alertmanager- p 9093root@sqm-docker01 9093-v / root/alertmanager.yml:/etc/alertmanager/alertmanager.yml-v / root/prometheus/alertmanager-tmpl:/etc/alertmanager-tmpl prom/alertmanager

Ensure that the container is functioning properly:

Next, stop a container to simulate downtime:

The status is down:

/ / QQ Mail received the alarm email:

/ / similarly, if your service returns to normal operation, an alarm email will be sent:

At this point, Prometheus monitoring has been deployed, and the mailbox alarm has been successfully realized.

-this is the end of this article. Thank you for reading-

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.