Shulou(Shulou.com)06/01 Report--

This article mainly introduces how Prometheus integrates AlertManager. It is very detailed and has a certain reference value. Friends who are interested must finish it!

Brief introduction

Alertmanager is mainly used to receive alarm information sent by Prometheus, it is easy to achieve alarm information de-duplication, noise reduction, grouping, policy routing, is an avant-garde alarm notification system. It supports rich alarm notification channels, which can forward alarm messages to mailboxes, WeCom, nails and so on.

Experimental preparation

Start the http-simulator Metrics Simulator:

Docker run-- name http-simulator-d-p 8080 pierrevincent/prom-http-simulator:0.1

Start Prometheus. In order to update the configuration, mount the configuration file:

Docker run-- name prometheus-d-p 9090 prom/prometheus 9090-v / Users/huanchu/Documents/prometheus-data:/prometheus-data\ prom/prometheus-- web.enable-lifecycle-- config.file=/prometheus-data/prometheus.yml

Startup adds the parameter-web.enable-lifecycle, which allows Prometheus to dynamically update the configuration through web endpoints.

Visit http://127.0.0.1:9090/targets, Prometheus's own metrics and http-simulator 's metrics are in the up state, and the preparations are done.

Experiment 1 alarm configuration

Under the prometheus-data folder, create the alarm profile simulator_alert_rules.yml:

Groups:- name: simulator-alert-rule rules:-alert: HttpSimulatorDown expr: sum (up {job= "http-simulator"}) = = 0 for: 1m labels: severity: critical

The configuration file means that when the http-simulator service up status is 0 and lasts for 1 minute, an alarm is generated at a level of "critical".

Modify the prometheus.yml and reference the simulator_alert_rules.yml file. The prometheus.yml content is as follows:

Global: scrape_interval: 5s evaluation_interval: 5s scrape_timeout: 5srule_files:-"simulator_alert_rules.yml" scrape_configs:-job_name: 'prometheus' static_configs:-targets: [' localhost:9090']-job_name: 'http-simulator' metrics_path: / metrics static_configs:-targets: [' 192.168.43.121 scrape_timeout 8080']

Update the Prometheus configuration:

Curl-X POST http://localhost:9090/-/reload

Visit http://127.0.0.1:9090/config and you can see that the configuration has been updated for:

A newly added alarm rule appears under access http://127.0.0.1:9090/rules dint rules:

Verification

When you visit http://127.0.0.1:9090/alerts, the HttpSimulatorDown under Alerts is green and is in INACTIVE status, indicating that nothing has happened.

Shut down the http-simulator service:

Docker stop http-simulator

Visiting http://127.0.0.1:9090/alerts Magnum HttpSimulatorDown turns yellow and is in PENDING state, indicating that the alarm is about to be activated.

A minute later, the HttpSimulatorDown turns red and is in the FIRING state, indicating that the alarm has been activated.

Lab 2 alarm configuration

Add alarm configuration to the simulator_alert_rules.yml file:

-alert: ErrorRateHigh expr: sum (rate (http_requests_total {job= "http-simulator", status= "500"} [5m])) / sum (rate (http_requests_total {job=" http-simulator "} [5m])) > 0.02 for: 1m labels: severity: major annotations: summary:" High ErrorRate detected "description:" ErrorRate is above 2% (current value is: {{$value}} ")

The configuration file means that when the error rate of the http-simulator request is 2% and lasts for 1 minute, an alarm is generated at a level of "very serious"

Update the Prometheus configuration:

Curl-X POST http://localhost:9090/-/reload verification

Visit http://127.0.0.1:9090/alerts MagneErrorRateHigh for the green INACTIVE state.

Adjust the error rate of http-simulator to 10%

Curl-H 'Content-Type: application/json'-X PUT-d'{"error_rate": 10} 'http://localhost:8080/error_rate

After a while, when you visit http://127.0.0.1:9090/alerts, you can see that the error rate has increased by 2%. The error rate is in red FIRING status, and the alarm has been activated.

Install and configure AlertManager

Install AlertManager by mounting files with docker, create a folder alertmanager-data folder locally, and create an alertmanager.yml in it, as follows:

Global: smtp_smarthost: 'smtp.163.com:25' smtp_from:' xxxxx@163.com' smtp_auth_username: 'xxxxx@163.com' smtp_auth_password:' xxxxx'route: group_interval: 1m # when the first alarm is sent, wait 'group_interval' time to send a new set of alarm messages repeat_interval: 1m # if an alarm message has been sent successfully Wait for 'repeat_interval' time to resend them receiver:' mail-receiver'receivers:- name: 'mail-receiver' email_configs:-to:' xxxxxx@163.com'

Start AlertManager:

Docker run-- name alertmanager-d-p 9093 Users/huanchu/Documents/alertmanager-data:/alertmanager-data Users/huanchu/Documents/alertmanager-data:/alertmanager-data\ prom/alertmanager-- config.file=/alertmanager-data/alertmanager.yml

Under the Prometheus directory, modify the prometheus.yml configuration Alertmanager address:

# Alertmanager configurationalerting: alertmanagers:-static_configs:-targets:-192.168.43.121

Update the Prometheus configuration:

Curl-X POST http://localhost:9090/-/reload

Visit http://127.0.0.1:9093, visit the Alertmanager UI interface, and you can see that the ErrorRateHigh alarm has been received:

The mailbox will receive an alarm message:

The above is all the content of the article "how Prometheus integrates AlertManager". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.