Shulou(Shulou.com)06/03 Report--

I. introduction to Prometheus

Prometheus is the earliest open source monitoring and alarm solution by SoundCloud. And has grown into the second CNCF graduate program after Kubernetes. With the popularization of cloud native concept and the development of Kubernetes and other technologies, Prometheus has made great progress in the field of monitoring. Its main components include Prometheus,Alertmanager,Node Exporter,Blackbox Exporter and Pushgateway.

Prometheus features:

Real-time monitoring. Provide hardware and software running status display alarm. In accordance with the preset alarm threshold, alarm information is sent in a variety of ways to assist decision-making. Big data monitoring not only provides real-time status display, but also helps fault backtracking and risk prediction.

II. Prometheus deployment

Reprint: https://blog.gmem.cc/prometheus-study-note

1. Installation environment:

Server1:centos7.6 master node 190.168.3.250

Server2:centos7.6 Node 1 190.168.3.251

Download the latest version of Prometheus for your platform, then unzip it and run it:

Https://prometheus.io/download/

Https://prometheus.io/docs/prometheus/latest/getting_started/

two。 Binary installation

Tar xvfz prometheus-2.15.0.linux-amd64.tar.gz

Cd prometheus-2.15.0.linux-amd64

Mv prometheus-2.15.0.linux-amd64 / usr/local/prometheus

Start

. / prometheus-- config.file=prometheus.yml

For convenience, add it to the service profile

Vi / usr/lib/systemd/system/prometheus.service

[Unit]

Description= https://prometheus.io

[Service]

Restart=on-failure

ExecStart=/usr/local/prometheus/prometheus-config.file=/usr/local/prometheus/prometheus.yml

[Install]

WantedBy=multi-user.target

Restart the service

Systemctl daemon-reload

Systemctl start prometheus.service

3. Container installation

Https://prometheus.io/docs/prometheus/latest/installation/

Prometheus.yml binds you from the host by running the following command:

Docker run-p 9090 9090-v / tmp/prometheus.yml:/etc/prometheus/prometheus.yml\

Prom/prometheus

Or use additional volumes for configuration:

Docker run-p 9090 9090-v / prometheus-data\

Prom/prometheus-config.file=/prometheus-data/prometheus.yml

4. Visit web

Http://190.168.3.250:9090 accesses its own status page

You can verify the metric of Prometheus itself by visiting localhost:9090: 190.168.3.250:9090/metrics

three。 Configure monitoring service

1. Configure Prometheus monitoring itself

Introduction to Global Profil

For a complete set of configuration options, see: https://prometheus.io/docs/prometheus/latest/configuration/configuration/

Prometheus periodically collects data from the monitoring target with scrape_interval rules, and then stores the data on local storage. Scrape_interval can be set globally or a single metrics.

Prometheus calculates the alarm rules periodically based on evaluation_interval rules, and then updates the alarm status. Evaluation_interval can only be set globally.

Global: global configuration

Alerting: alarm configuration

Rule_files: alarm rules

Scrape_configs: configure the data source, called target, and each target is named after job_name. It is divided into static configuration and service discovery

Global:

Default crawl cycle, which can be set in units ms and smhdwy # to collect data every 15 seconds. Default is 1 minute.

[scrape_interval: | default = 1m]

Default crawl timeout

[scrape_timeout: | default = 10s]

The default period of the estimation rule # calculates the rule every 15 seconds. Default 1 minute

[evaluation_interval: | default = 1m]

A list of tags that are forced to be added for time series or Alert when communicating with an external system, such as AlertManager

External_labels:

[:...]

List of rule files

Rule_files:

[-.]

Crawl configuration list

Scrape_configs:

[-.]

Alertmanager related configuration

Alerting:

Alert_relabel_configs:

[-.]

Alertmanagers:

[-.]

Configuration related to remote read and write features

Remote_write:

[-.]

Remote_read:

[-.]

Vi prometheus.yml

The following is the data configuration for pulling your own service sampling points

Scrape_configs:

Do not monitor the metrics, the job name will be added to all the sampling points pulled, and a host:port tag of the instance target service will also be added to the sampling points.

Job_name: 'prometheus'

Cover the sampling points of global, and pull time interval of 5s

Scrape_interval: 5s

Static_configs:targets: ['localhost:9090']

At the bottom, statically configure to monitor the machine and collect data from port 9090 of the machine.

Note: use promtool to check whether the configuration file is correct each time the configuration is modified

[root@server1 prometheus] #. / promtool check config prometheus.yml

Restart the service

You can use the kill-hup process id to automatically load new configuration files

Check targets and you can see that the node is normal.

You can also search for the collected data here.

two。 Configure service discovery to monitor linux hosts and related services

Install node_exporter on server2 190.168.3.251

Working with documents: https://prometheus.io/docs/guides/node-exporter/

GitHub: https://github.com/prometheus/node_exporter

Exporter list: https://prometheus.io/docs/instrumenting/exporters/

Install node collector, mysql service, nginx service in server2

Tar zxf node_exporter-0.17.0.linux-amd64.tar.gz

Mv node_exporter-0.17.0.linux-amd64 / usr/local/node_exporter

Cd / usr/local/node_exporter/

. / node_exporter-- help

Service discovery, we use file_sd_configs here, write monitoring configuration files, and service discovery node nodes

Mkdir sd_config creates a service discovery directory

Write acquisition address targets

[root@server1 prometheus] # vi sd_config/node.yml

It is convenient to manage the service. Add the host node monitoring collection node_exporter to the service configuration file.

If you want to monitor the system service of a node, you need to add list parameters later

-- collector.systemd.unit-whitelist= ". +" loops the regular matching unit from systemd

-- collector.systemd.unit-whitelist= "(docker | sshd | nginx) .service" whitelist, collection target

/ usr/bin/node_exporter-collector.systemd-collector.systemd.unit-whitelist= (docker | sshd | nginx) .service

In http://190.168.3.250:9090/graph, you can already see the host information collected by the node, and you can already see that the service is monitored. A value of 1 means that the service is normal.

Add:

The static collection of 9090 monitored on the previous 190.168.3.250 can also be changed into the form of file service discovery.

[root@server1 prometheus] # vi sd_config/test.yml, some of which are tagged and the collection address is added, which can be found by using tags in targets

four。 Use the grafana front end to display data and monitor docker services

1.cAdvisor collects container information

Use cAdvisor (Container Advisor) to collect running container resource usage and performance information.

Docker run\

-- volume=/:/rootfs:ro\

-- volume=/var/run:/var/run:ro\

-- volume=/sys:/sys:ro\

-- volume=/var/lib/docker/:/var/lib/docker:ro\

-- volume=/dev/disk/:/dev/disk:ro\

-- publish=8080:8080\

-- detach=true\

-- name=cadvisor\

Http://190.168.3.250:8080/ accesses the web port and has collected data from the container

2.Grafana

Https://grafana.com/grafana/download

Grafana is an open source measurement analysis and visualization system. Grafana supports query Prometheus. Since Grafana 2.5.0 (2015-10-28), Grafana data sources for Prometheus have been included.

Download https://grafana.com/grafana/download?platform=docker on the official website

We use the docker version here.

Docker run-d-- name=grafana-p 3000UR 3000 grafana/grafana

After running, 190.168.3.250 web 3000 accesses the interface

Password accounts are all admin. You need to change your password the first time you log in.

Add data source

There is already a Prometheus data source

3. Add a monitoring host template

Enter 9276, import

Modify the name and database

Finish, view the final template effect

4. View docker Monitoring

Add a new job and modify the collection configuration file prometheus.yml

[root@server1 prometheus] # vi prometheus.yml

After checking the configuration file, restart the main service. / promtool check config prometheus.yml

The same method is used to download the template at Grafana and import 193s.

You can also download other templates at https://grafana.com/dashboards/

five。 Monitor mysql servic

Mysql_exporter: used to collect MySQL performance information.

Download https://github.com/prometheus/mysqld_exporter

Monitoring template https://grafana.com/dashboards/7362

We configure it on server2 190.168.3.251 and need to download the mysqld_exporter database to collect and install mariadb

Log in to mysql to create an account for exporter so that you can collect data:

Mysql > CREATE USER 'exporter'@'localhost' IDENTIFIED BY' 123456'

Mysql > GRANT PROCESS, REPLICATION CLIENT, SELECT ON. TO 'exporter'@'localhost'

Vi .my.cnf, add the hidden database account of the user, which is required for prometheus monitoring mysql.

Modify the collection configuration file in server1 190.168.3.250

[root@server1 prometheus] # vi prometheus.yml

Check the configuration file and restart the service, systemctl restart prometheus.service

[root@server2 mysql_exporter] #. / mysqld_exporter-config.my-cnf=.my.cnf

The database information has been collected.

Http://190.168.3.251:9104/metrics

After importing the template after 7362

six。 Mail alarm

1. Alarm introduction

In the Prometheus platform, alerts are handled by a separate component, Alertmanager. Typically, we first tell Prometheus Alertmanager where it is, then create an alert rule in the Prometheus configuration, and finally configure Alertmanager to process the alert and send it to the recipient (email, webhook, slack, etc.).

Address 1: https://prometheus.io/download/

Address 2: https://github.com/prometheus/alertmanager/releases

The main steps for setting alarms and notifications are as follows:

The first step to deploy Alertmanager

The second step is to configure Prometheus to communicate with Alertmanager

The third step is to create alarm rules in Prometheus

two。 Configure Prometheus to communicate with Alertmanager

You need to download the above alarm package on the official website.

Modify the alarm media and send it using mailbox

[root@server1 alertmanager-0.16.0.linux-amd64] # vi alertmanager.yml

Note: the above authentication password is not an email password, but an authorization code.

After modification, check that the configuration file alertmanager.yml is correct, and then restart the service

. / alertmanager-- config.file alertmanager.yml

Systemctl restart prometheus.service

Modify prometheus connection alarm module and set alarm rules

[root@server1 prometheus] # vi prometheus.yml

[root@server1 prometheus] # vi rules/test.yml

Refer to the alarm on the official website.

Groups:

Name: example # alarm rule group name

Rules:alert: InstanceDown

Expr: up = = 0

For: 1m # duration, indicating that no information can be obtained for one minute, then an alarm will be triggered.

Labels:

Severity: page # Custom tag

Annotations:

Summary: "Instance {{$labels.instance}} down" # Custom Summary

Description: "{{$labels.instance}} of job {{$labels.job}} has been down for more than 5 minutes." # Custom specific description

3. Test mailbox alarm

Let's turn off the mysql monitor on server2 and test the alarm.

Http://190.168.3.250:9090/alerts, has been triggered

We checked our mailbox and got a call.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.