Shulou(Shulou.com)06/03 Report--

Blog outline:

Deploy Weave- Weave- II, start configuration 1, run Node Server Container 2, run cAdvisor Container 3, run Prometheus server Container 4 on docker01, run grafana Container 5 on docker01, and set Prometheus alarm

Prometheus is a system and service monitoring system. It collects metrics from configured targets at a given interval, evaluates rule expressions, displays results, and triggers alerts when certain conditions are found to be true.

Compared with other monitoring systems, the main distinguishing features of Prometheus are:

A multidimensional data model (time series defined by indicator names and setting key / value dimensions) A flexible query language to take advantage of this dimension does not depend on distributed storage; single server nodes are autonomous time series collection through the pull model on HTTP to support push time series through intermediate gateways through service discovery or static configuration to discover targets multiple graphics and dashboards support hierarchical and horizontal federation

The official diagram of the architecture is as follows:

Deploy the service, including four components: Prometheus Server, Node Exporter, cAdvrisor, and Grafana.

The functions of each component are as follows:

Master server of Prometheus Server:Prometheus service; Node Exporter: collect information about Host hardware and operating system; cAdvrisor: collect container information running on Host; Grafana: used to show Prometheus monitoring operation interface (provide us with a friendly web interface).

For all the descriptions of the above four components, please go to the Github official website and directly search for the corresponding docker image name (in the following deployment process, each container running a service will specify its image name, which can be searched by referring to the image name in the command). You can find a detailed description of the components, which is not discussed here.

I. Environmental preparation

My environment here is as follows:

2. Start configuration 1. Run Node Server container

The component needs to run on all hosts that need to be monitored, that is, all three of my servers need to execute the following command to run the container component.

[root@docker01 ~] # docker run-d-p 9100sys:/host/sys 9100-v / proc:/host/proc-v / sys:/host/sys-v /: / rootfs-- net=host-- restart=always prom/node-exporter-- path.procfs / host/proc-- path.sysfs / host/sys-- collector.filesystem.ignored-mount-points "^ / (sys | proc | dev | host | etc) ($| /)" # based on the "prom/node-exporter" image operation container, you can search for the image on github's official website. In order to understand its main functions # Note: each host that needs to be monitored needs to execute the above command to run the container in order to collect host information

After each server runs the above command, the browser accesses the IP address of the docker server + 9100 port, and you can see the following interface, which indicates that the container is running well. It doesn't matter if you don't understand the content, the information is not for us to read, we are looking at the web interface provided to us at last. The web page you see is as follows:

It's a good idea to visit port 9100 of all the servers running the above containers to make sure you can see the above page.

2. Run the cAdvisor container

CAdvrisor is responsible for collecting container information running on Host. Similarly, run the cAdvisor container by executing the following command on all servers that need to be monitored:

[root@docker01] # docker run-v /: / rootfs:ro-v / var/run:/var/run:rw-v / sys:/sys:ro-v / var/lib/docker:/var/lib/docker:ro-p 8080pur808080-- detach=true-- name=cadvisor-- net=host google/cadvisor

After each server runs the above command, the browser accesses the server's IP address + 8080 port and you will see the following page:

Similarly, it is best to access port 8080 of each server to confirm success.

3. Run the Prometheus server container on docker01

Prometheus Server is the primary server, so you only need to run this container on one of them. Here I run it on the docker01 server:

[root@docker01] # docker run-d-p 9090 name prometheus-- net=host prom/prometheus# first run any container based on the prom/prometheus image We need to make a copy of its main configuration file to make changes [root@docker01 ~] # docker cp prometheus:/etc/prometheus/prometheus.yml / root/# copy the main configuration file in the prometheus container to the host local [root@docker01 ~] # docker rm-f prometheus # Delete the newly created container and [root@docker01 ~] # vim prometheus.yml # Open the copied configuration file Jump directly to the last line of the configuration file. # omit part of the content # modify as follows:-targets: ['localhost:9090','localhost:8080','localhost:9100','192.168.20.7:9100','192.168.20.7:8080','192.168.20.8:9100' '192.168.20.8purl 8080'] # the above seems to be disorganized In fact, it only specifies the local ports 9090, 8080 and 9100, and # adds ports 8080 and 9100 of the other two monitored servers. If you need to monitor more servers, just specify and add them one by one. Of course, the monitored end needs to run the previous two containers # after the modification is completed. Save and exit [root@docker01 ~] # docker run-d-p 9090 name prometheus 9090-v / root/prometheus.yml:/etc/prometheus/prometheus.yml-- name prometheus-- net=host prom/prometheus# execute the above command, run the new prometheus container, and mount the modified master configuration file to the specified location in the container. # if you want to modify the main configuration file later, modify the local one directly. # after the master configuration file is mounted, the local one is the same as that in the container. If the content is modified locally, it will be synchronized to the container.

When the client accesses port 9090 of docker01, you will see the following page:

Then click "status" above, and then click "Targets", as shown in the following figure:

When you see the following page, it indicates that everything you have done so far is all right:

4. Run the grafana container on docker01

This container provides us with a friendly web presentation page.

[root@docker01 ~] # mkdir grafana-storage [root@docker01 ~] # chmod 777-R grafana-storage/ [root@docker01 ~] # docker run-d-p 3000 docker run-d-p 3000-name grafana- v / root/grafana-storage:/var/lib/grafana-e "GF_SECURITY_ADMIN_PASSWORD=123.com" grafana/grafana# the "- e" option in the above command is to set the default login user admin with a password of "123.com". # if iptables and other related error messages are prompted during the startup of the container, # you need to execute the command systemctl restart docker, restart the docker service, and then rerun the container # but it should be noted that if the "--restart=always" option is not added when running the container, # then all containers need to be restarted manually after restarting the docker service. # restart all containers command "docker ps-a-Q | xargs docker start"

After the container is running, you can use the client to access the docker01IP address + port 3000, and you can see the following page:

On the login page above, enter the user name "admin", the password is the password we specified at run time, and here I am "123.com". Log in after typing, you will see the following interface, and then click add data Source:

See the interface below, find Prometheus, and click "select" to select it:

Modify the configuration according to the prompts given in the following figure:

After the above configuration is completed, we need to configure it in what form to show us that it can be customized, but it is very troublesome. I choose to go directly to the grafana official website to find a ready-made template.

Log in to the grafana website and click the following options:

Look at the picture and operate:

There are two ways to import these templates into our grafana web interface:

Method 1:

1) after entering the template, click "Download JSON" to download:

2) after downloading, go back to the grafana interface and click as follows:

3) Click "Upload.json file", and then upload the template we downloaded from the grafana official website:

4) after uploading, please see the following figure to perform the operation:

5) at this point, you can see the following monitoring page, indicating that the import is successful:

However, if you look at it, you will find that some information about this template can not be detected, so here is just to show the first way to import the template, I prefer to use method 2 for import.

Mode 2

1) after entering the template provided on the official website, we need to record its ID number, as follows:

2) then go back to the web interface of grafana, and click the following to import:

3) enter our recorded ID number in the red box below:

4) the next operation is the same as in method 1, as follows:

5) after importing this template, we can see the interface provided by the template, as follows:

At this point, the monitoring of the web interface is complete.

5. Set Prometheus alarm

There are several ways to alarm Prometheus, such as mailbox, nail, Wechat and so on. I choose the alarm method of mailbox here.

1) run alertmanager container [root@docker01 ~] # docker run on the docker01 server-- name alertmanager-d-p 9093 prom/alertmanager# simply run a container [root@docker01 ~] # docker cp alertmanager:/etc/alertmanager/alertmanager.yml / root# to copy the configuration files in the container to the local [root@docker01 ~] # docker rm-f alertmanager# container [root@docker01 ~] # vim alertmanager.yml # before deletion The edited configuration file # configuration file can be divided into the following groups: # global: global configuration. Set alarm policies, alarm channels, etc.; # route: distribution policy; # receivers: recipient, specify who will receive the messages you send; # inhibit_rules: suppression policy. When there is another set of matching alerts, the suppression rule is disabled on a set of matching alerts.

The changed configuration file here is as follows:

[root@docker01 ~] # vim alertmanager.yml # Edit the copied configuration file # change all the following email accounts and auth_password to your own account global: resolve_timeout: 5m smtp_from: '916551516roomq.com' smtp_smarthost: 'smtp.qq.com:465' smtp_auth_username:' 916551516roomqq.com 'smtp_auth_password:' abdgwyaxickabccb0' smtp_require_tls: false smtp _ hello: 'qq.com'route: group_by: [' alertname'] group_wait: 5s group_interval: 5s repeat_interval: 5m receiver: 'email'receivers:- name:' email' email_configs:-to: '916551516roomqq.com' send_resolved: true When the container returns to normal, it will also send an email inhibit_rules:-source_match: severity: 'critical' target_match: severity:' warning' equal: ['alertname',' dev', 'instance'] # after the change is completed Save and exit [root@docker01 ~] # docker run-d-- name alertmanger-p 9093 name alertmanger-p 9093-v / root/alertmanager.yml:/etc/alertmanager/alertmanager.yml-restart=always prom/alertmanager# to run the new alertmanager container and mount the changed configuration file # if there is something wrong with the configuration file, then this container will not work. 2) set alertmanager alarm rules [root@docker01 ~] # mkdir-p prometheus/rules [root@docker01 ~] # cd prometheus/rules/ [root@docker01 rules] # vim node-up.rules # Editing rules are as follows: groups:- name: node-up # set the name of the alarm rules:-alert: node-up expr: up {job= "prometheus"} = 0 # the job must be exactly the same as job_name in Prometheus's configuration file For: 15s labels: severity: 1 # first level warning team: node annotations: summary: "{{$labels.instance}} has stopped running for more than 15s!" # after the definition is completed, save and exit.

If you want to write your own alarm rules, you can refer to its official documents. There are some problems with my alarm rules here, but you can receive its alarm messages.

3) Associate to Prometheus [root @ docker01 ~] # vim prometheus.yml # change the following content: alertmanagers:-static_configs:-targets:-192.168.20.6 static_configs 9093 # remove the comment on this line and change it to the IP+ port of the alertmanager container rule_files:-"/ usr/local/prometheus/rules/*.rules" # Note that this path is the path within the container # you must pay special attention to the format of the configuration file Pay attention to indentation. After the changes are complete, save and exit.

To prevent format errors, I attach a screenshot of the configuration file:

[root@docker01 ~] # docker rm-f prometheus# Delete this container [root@docker01 ~] # docker run-d-p 9090 prometheus# 9090-v / root/prometheus.yml:/etc/prometheus/prometheus.yml-v / root/prometheus/rules:/usr/local/prometheus/rules-- name prometheus-- net=host prom/prometheus# rerun this container, mount a new file [root@docker01 ~] # docker logs prometheus# if you encounter an error in startup, you can check the container's log error

At this point, if the target in the Prometheus page has a container dropped by down, an alarm message will be sent to your mailbox. When the container is normal, it will also give you feedback.

The alarm email I received is as follows:

4) set alarm message template [root@docker01 ~] # cd prometheus/ [root@docker01 prometheus] # mkdir alertmanager-tmpl [root@docker01 prometheus] # cd alertmanager-tmpl/ [root@docker01 alertmanager-tmpl] # vim email.tmpl # Edit alarm template {{define "email.from"}} 916551516@qq.com {{end}} # change this to your mailbox {{define "email.to"}} 916551516@qq.com {{end}} # change to your own mailbox {{define "email.to.html"}} {{range .Alerts}} = start=

Alarm program: prometheus_alert

Alarm level: {{.Labels.alarm}} level

Alarm type: {{.Labels.alertname}}

Failed host: {{.Labels.instance}}

Alarm topic: {{.Annotations.summary}}

Trigger time: {{.StartsAt.Format "2006-01-02 15:04:05"}}

= end=

{{end}} {{end}} # after changing your mailbox, you can save and exit [root@docker01 ~] # cd [root@docker01 ~] # vim alertmanager.yml # to edit the alertmanager configuration file Modify the following annotated line global: resolve_timeout: 5m smtp_from: '916551516roomqq.com' smtp_smarthost:' smtp.qq.com:465' smtp_auth_username: '916551516roomqq.com' smtp_auth_password:' abdgwyaxickabccb0' smtp_require_tls: false smtp_hello: 'qq.com'templates: # add this line -' / etc/alertmanager-tmpl/*.tmpl' # add this line route: group_by: ['alertname'] group_wait: 5s group_interval: 5s repeat_interval: 5m receiver:' email'receivers:- name: 'email' email_configs:-to:' {template "email.to}}'# must be the corresponding html in {{define" email.to "} 916551516@qq.com {{end}}:'{template" email.to.html. }}'# must correspond to the name in {{define "email.to.html"}} send_resolved: trueinhibit_rules:-source_match: severity: 'critical' target_match: severity:' warning' equal: ['alertname',' dev', 'instance'] # after the modification is completed Save exit [root@docker01 ~] # docker rm-f alertmanger # Delete this container [root@docker01 ~] # docker run-d-- name alertmanger-p 9093alertmanger 9093-v / root/alertmanager.yml:/etc/alertmanager/alertmanager.yml-v / root/prometheus/alertmanager-tmpl:/etc/alertmanager-tmpl-- restart=always prom/alertmanager# run a new container [root@docker01 ~] # docker ps # confirm that port 9093 is running mapping normal CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESf9d718071d4d prom/alertmanager "/ bin/alertmanager-…" 7 seconds ago Up 6 seconds 0.0.0.0 9093/tcp alertmanger1b1379e3e12a prom/prometheus 9093-> 9093/tcp alertmanger1b1379e3e12a prom/prometheus "/ bin/prometheus-c …" 11 minutes ago Up 11 minutes prometheus65240a28e383 grafana/grafana "/ run.sh" 2 hours ago Up 2 hours 0.0.0.0 hours ago Up 3000-> 3000/tcp grafana3bd83eb145e2 google/cadvisor "/ usr/bin/cadvisor-…" 2 hours ago Up 2 hours cadvisora57b77a33e79 prom/node-exporter "/ bin/node_exporter …" 2 hours ago Up 2 hours eloquent_wescoff

At this point, a new alarm template is also generated. If the following container has Down, a new email will be sent to you, and when it returns to normal, it will also be sent. Similarly, the content of the email is in the wrong format, but you can receive the alarm message normally. If you want to change the alarm template, you can refer to the github official documentation.

The alarm message I received in my mailbox is as follows:

-this is the end of this article. Thank you for reading-

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.