Shulou(Shulou.com)06/03 Report--

Related recommendation

1. Monitoring kubetnetes clusters using Prometheus Operator

two。 Using Prometheus Operator to implement Application Custom Monitoring

I. Overview

Alertmanager and Prometheus are two separate components. The Prometheus server sends alerts to Alertmanager according to the alarm rules, and then Alertmanager sends notifications to silencing, inhibition, aggregation and other messages via email, dingtalk and HipChat.

Alertmanager handles alerts from clients such as Prometheus servers. It is responsible for removing duplicates, packets, and routing alerts to the correct receivers, such as email, Slack, dingtalk, and so on. Alertmanager also supports groups,silencing and alarm suppression mechanisms.

Nail as an internal communication tool, basically everyone can be used on computers and mobile phones, the message can be checked at the first time, the immediate requirements of the alarm message is relatively high, so it is suitable to use nail notification.

Second, add nailing robot

Please refer to the official document: custom Robot

Get the hook of the robot after adding the robot (it seems that the robot can only be added in the nail group), which will be used in the later deployment.

Robot hook: https://oapi.dingtalk.com/robot/send?access_token=xxxxxx

3. Configure Alertmanager

Alertmanager official document: https://github.com/prometheus/docs/blob/db2a09a8a7e193d6e474f37055908a6d432b88b5/content/docs/alerting/configuration.md#webhook_config

Modify the Alertmanager alarm configuration, because the above official documentation has given the details of each parameter, it will not be explained one by one.

[root@node-01 prometheus] # vim prometheus-operator/values.yaml config: global: resolve_timeout: 2m route: group_by: ['job'] group_wait: 30s group_interval: 2m repeat_interval: 12h receiver:' webhook' routes:-match: alertname: DeadMansSwitch receiver: 'webhook' receivers:-name:' webhook' Webhook_configs:-url: http://webhook-dingtalk/dingtalk/send/ send_resolved: true

Update prometheus-operator

[root@node-01 prometheus] # helm upgrade p. / prometheus-operator

After successful modification, you can see the relevant configuration on the status page of alertmanager.

IV. Convert alertmanager data format

Alertmanager sends the following data in JSON format to the endpoint through an HTTP POST request:

{"version": "4", "groupKey":, / / key identifying the group of alerts (e.g. To deduplicate) "status": "," receiver ":," groupLabels ":," commonLabels ":," commonAnnotations ":," externalURL ":, / / backlink to the Alertmanager. "alerts": [{"labels":, "annotations":, "startsAt": "," endsAt ":"},...]}

This is an example of testing alarm data:

B'{"receiver": "webhook", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "DeadMansSwitch", "prometheus": "monitoring/p-prometheus", "severity": "none"}, "annotations": {"message": "This is a DeadMansSwitch meant to ensure that the entire alerting pipeline is functional."} "startsAt": "2019-03-08T10:02:28.680317737Z", "endsAt": "0001-01-01T00:00:00Z", "generatorURL": "http://prom.cnlinux.club/graph?g0.expr=vector%281%29\\u0026g0.tab=1"}],"groupLabels":{},"commonLabels":{" alertname ":" DeadMansSwitch "," prometheus ":" monitoring/p-prometheus "," severity ":" none "} "commonAnnotations": {"message": "This is a DeadMansSwitch meant to ensure that the entire alerting pipeline is functional."}, "externalURL": "http://alert.cnlinux.club","version":"4","groupKey":"{}/{alertname=\\"DeadMansSwitch\\"}:{}"}\n'

Nailing has a requirement on the format of the data (specifically, the official document is required to be nailed on it), so the data sent by Alertmanager needs to be converted into format.

Below we use our own python script to convert.

Script description:

Among the data sent by alertmanager, what is important is the data of labels {}, but there is too much data in it, and a lot of information is not needed in the alarm information, so an EXCLUDE_LIST list is added in the script to exclude unwanted data. [root@node-01 prometheus] # cat app.pyknoxxinxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx "en_US.UTF-8") app = Flask (_ name__) console = logging.StreamHandler () fmt ='% (asctime) s -% (filename) s name% (lineno) s -% (message) s'formatter = logging.Formatter (fmt) console.setFormatter (formatter) log = logging.getLogger ("flask_webhook_dingtalk") log.addHandler (console) log.setLevel (logging.DEBUG) EXCLUDE_LIST = ['prometheus' " 'endpoint'] @ app.route (' /') def index (): return 'Webhook Dingtalk by Billy https://blog.51cto.com/billy98'@app.route('/dingtalk/send/', Methods= ['POST']) def hander_session (): profile_url = sys.argv [1] post_data = request.get_data () post_data = json.loads (post_data.decode ("utf-8")) [' alerts'] post_data = post_data [0] messa_list = [] messa_list.append ('# alarm type:% s'% post_data ['status']. Upper () messa_list.append ('* * startsAt:**% s'% post_data ['startsAt']) for i in post_data [' labels'] .keys (): if i in EXCLUDE_LIST: continue else: messa_list.append ("* *% s"% (I) Post_data ['labels'] [I]) messa_list.append (' * Describe:**% s'% post_data ['annotations'] [' message']) messa = ('\\ n\ n > '.join (messa_list)) status = alert_data (messa, post_data [' labels'] ['alertname'], profile_url) log.info (status) return statusdef alert_data (data,title Profile_url): headers = {'Content-Type':'application/json'} send_data =' {"msgtype": "markdown", "markdown": {"title":\ "% s\", "text":\ "% s\"}'% (title,data) # type: str send_data = send_data.encode ('utf-8') reps = requests.post (url=profile_url, data=send_data) Headers=headers) return reps.textif _ _ name__ = ='_ main__': app.debug = False app.run (host='0.0.0.0', port='8080') 5. Make a Docker image

Mirror the above python scripts and run them in the K8s cluster as a service to ensure high availability.

You can also use the image I have made successfully: docker pull billy98/webhook-dingtalk:latest, just pull it.

[root@node-01 prometheus] # cat DockerfileFROM centos:7 as buildMAINTAINER billy98 5884625@qq.comRUN curl-o / etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo & & yum install-y python36 python36-pip & & pip3.6 install flask requests werkzeugADD app.py / usr/local/alert-dingtalk.pyFROM gcr.io/distroless/python3COPY-- from=build / usr/local/alert-dingtalk.py / usr/local/alert-dingtalk.pyCOPY-- from=build usr / local/lib64/python3.6/site-packages usr/local/lib64/python3.6/site-packagesCOPY-- from=build usr/local/lib/python3.6/site-packages usr/local/lib/python3.6/site-packagesENV PYTHONPATH=usr/local/lib/python3.6/site-packages:usr/local/lib64/python3.6/site-packagesEXPOSE 8080ENTRYPOINT ["python" "/ usr/local/alert-dingtalk.py"] [root@node-01 prometheus] # docker build-t billy98/webhook-dingtalk:latest.

The image from my build is only more than 50 MB. For more information on how to use it, please see:

Distroless: https://github.com/GoogleContainerTools/distroless

VI. Deployment of webhook-dingtalk [root@node-01 prometheus] # cat webhook-dingtalk.yaml apiVersion: apps/v1beta2kind: Deploymentmetadata: labels: app: webhook-dingtalk name: webhook-dingtalk namespace: monitoring # is required in the same namespacespec: replicas: 1 selector: matchLabels: app: webhook-dingtalk template: metadata: labels: app: webhook-dingtalk spec: containers:-image: billy98/webhook-dingtalk:latest as alertmanager Name: webhook-dingtalk args:-"the nailing robot created above https://oapi.dingtalk.com/robot/send?access_token=xxxxxx" # hook ports:-containerPort: 8080 protocol: TCP resources: requests: cpu: 100m memory: 100Mi limits: cpu: 500m Memory: 500Mi livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 tcpSocket: port: 8080 readinessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 httpGet: port: 8080 Path: / imagePullSecrets:-name: IfNotPresent---apiVersion: v1kind: Servicemetadata: labels: app: webhook-dingtalk name: webhook-dingtalk namespace: monitoring # needs to be in the same namespacespec: ports:-name: http port: 80 protocol: TCP targetPort: 8080 selector: app: webhook-dingtalk type: ClusterIP as alertmanager

The alarm message in the nail is as follows:

Alarm recovery message

So far, all the operations have been completed.

If you have any questions, please leave a message below. I hope you will pay more attention and like it. Thank you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.