Shulou(Shulou.com)06/03 Report--

Overview of three-step installation of kubernetes Cluster

Prometheus operator should be the best practice for using the monitoring system. First of all, it builds the entire monitoring system with one click and configures it through some non-invasive means such as monitoring data sources.

Automatic fault recovery, highly available alarms, etc.

However, there is still a small threshold for beginners to use. This article combines the example of how to monitor envoy to share the correct posture of using prometheus operator.

As for how to write alarm rules and how to configure prometheus query statements is not the focus of this article, will be shared in subsequent articles, this article focuses on how to use prometheus operator

Prometheus operator installation

Prometheus operator is already included in the offline installation package of sealyun. You can use it directly after installation.

Configure monitoring data source

Principle: discover the monitoring data source service through operator's CRD

Start envoyapiVersion: apps/v1kind: Deploymentmetadata: name: envoy labels: app: envoyspec: replicas: 1 selector: matchLabels: app: envoy template: metadata: labels: app: envoyspec: volumes:-hostPath: # mount the envory configuration file path: / root/envoy type: DirectoryOrCreate name: envoy containers: -name: envoy volumeMounts:-mountPath: / etc/envoy name: envoy readOnly: true image: envoyproxy/envoy:latest ports:-containerPort: 10000 # data port-containerPort: 9901 # management port Metric is exposed through this port-kind: ServiceapiVersion: v1metadata: name: envoy labels: app: envoy # tag service, operator will look for this servicespec: selector: app: envoy ports:-protocol: TCP port: 80 targetPort: 10000 name: user-protocol: TCP # service expose metric port port: 81 targetPort: 9901 name: metrics # name is very important, ServiceMonitor will find the port name

Envoy profile:

The listening address must be changed to 0.0.0.0, otherwise the metric cannot be obtained through service

/ root/envoy/envoy.yaml

Admin: access_log_path: / tmp/admin_access.log address: socket_address: protocol: TCP address: 0.0.0.0 # here must be changed to 0.0.0.0 It cannot be 127.0.0.1 port_value: 9901static_resources: listeners:-name: listener_0 address: socket_address: protocol: TCP address: 0.0.0.0 port_value: 10000 filter_chains:-filters:-name: envoy.http_connection_manager config: stat_prefix: ingress_http route_config: Name: local_route virtual_hosts:-name: local_service domains: ["*"] routes:-match: prefix: "/" route: host_rewrite: sealyun.com cluster: service_google http _ filters:-name: envoy.router clusters:-name: service_sealyun connect_timeout: 0.25s type: LOGICAL_DNS # Comment out the following line to test on V6 networks dns_lookup_family: V4_ONLY lb_policy: ROUND_ROBIN hosts:-socket_address: address: sealyun.com port_value: 443 tls_context: {sni: sealyun.com} use ServiceMonitor

EnvoyServiceMonitor.yaml:

ApiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata: labels: app: envoy name: envoy namespace: monitoring # this port name can be different from service spec: endpoints:-interval: 15s port: metrics # envoy service path: / stats/prometheus # data source path namespaceSelector: matchNames: # envoy service namespace-default selector: matchLabels: app: envoy # Select envoy service

After the success of create, we can see the data source of envoy:

Then you can see metric:

Then you can do some configuration on grafana. The use of promethues is not the object discussed in this article.

Alarm configuration alertmanager configuration [root@dev-86-201 envoy] # kubectl get secret-n monitoringNAME TYPE DATA AGEalertmanager-main Opaque 1 27d

We can see this secrect and take a look at the details:

[root@dev-86-201envoy] # kubectl get secret alertmanager-main-o yaml-n monitoringapiVersion: v1data: alertmanager.yaml: Imdsb2JhbCI6IAogICJyZXNvbHZlX3RpbWVvdXQiOiAiNW0iCiJyZWNlaXZlcnMiOiAKLSAibmFtZSI6ICJudWxsIgoicm91dGUiOiAKICAiZ3JvdXBfYnkiOiAKICAtICJqb2IiCiAgImdyb3VwX2ludGVydmFsIjogIjVtIgogICJncm91cF93YWl0IjogIjMwcyIKICAicmVjZWl2ZXIiOiAibnVsbCIKICAicmVwZWF0X2ludGVydmFsIjogIjEyaCIKICAicm91dGVzIjogCiAgLSAibWF0Y2giOiAKICAgICAgImFsZXJ0bmFtZSI6ICJEZWFkTWFuc1N3aXRjaCIKICAgICJyZWNlaXZlciI6ICJudWxsIg==kind: Secret

Base64 decode it:

"global": "resolve_timeout": "receivers":-"name": "null"route": "group_by":-"job"group_interval": "5m"group_wait": "30s"receiver": "null"repeat_interval": "12h"routes":-"match": "alertname": "DeadMansSwitch"receiver": "null"

So configuring alertmanager is very simple, just create a secrect.

For example, alertmanager.yaml:

Global: smtp_smarthost: 'smtp.qq.com:465' smtp_from:' 4785153 roomqq.com 'smtp_auth_username:' 4785153roomqq.com 'smtp_auth_password:' xxx' # this password is generated after smtp authorization is enabled. Here is how to configure smtp_require_tls: falseroute: group_by: ['alertmanager','cluster' 'service'] group_wait: 30s group_interval: 5m repeat_interval: 3h receiver:' fanux' routes:-receiver: 'fanux'receivers:- name:' fanux' email_configs:-to: '474785153roomqq.com send_resolved: true

Delete can replace the old secret and regenerate the secret according to its own configuration.

Kubectl delete secret alertmanager-main-n monitoringkubectl create secret generic alertmanager-main-- from-file=alertmanager.yaml-n monitoring mailbox configuration, taking QQ Mail as an example

Enable the smtp pop3 service

Just follow the operation, and then an authorization code will pop up and configure it to the configuration file above.

Then you can receive an alarm:

Alarm rule configuration

Prometheus operator customizes PrometheusRule crd to describe alarm rules

[root@dev-86-202 shell] # kubectl get PrometheusRule-n monitoringNAME AGEprometheus-k8s-rules 6m

You can directly edit this rule, or you can create a PrometheusRule yourself.

Kubectl edit PrometheusRule prometheus-k8s-rules-n monitoring

If we add an alarm to group:

Spec: groups:-name:. / example.rules rules:-alert: ExampleAlert expr: vector (1)-name: k8s.rules rules:

Restart prometheuspod:

Kubectl delete pod prometheus-k8s-0 prometheus-k8s-1-n monitoring

Then you can see the new rules on the interface:

Discussion on Additive QQ Group: 98488045

Official account:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.