Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about how to configure custom alarm rules in Prometheus. Many people may not know much about it. In order to make you understand better, the editor has summarized the following for you. I hope you can get something from this article.

Understand Prometheus and its abstract concepts

From the following figure, we will see all the components that make up the Prometheus ecology:

The following are the terms related to this article, which you can quickly understand:

Prometheus Server: the main component of grabbing and storing metrics in a temporal database

Fetching: a method of pulling to obtain indicators. It is usually crawled at intervals of 10-60 seconds.

Target: the server client that retrieves data

Service discovery: enable Prometheus to identify the applications it needs to monitor and pull metrics in a dynamic environment

Alert Manager: the component responsible for handling alarms (including silencing, inhibition, aggregating alarm information, and sending alarm notifications by email, PagerDuty, Slack, etc.).

Data visualization: the captured data is stored in local storage and queried directly using PromQL, or viewed through Grafana dashboard.

Understand Prometheus Operator

According to CoreOS, the project owner of Prometheus Operator, Prometheus Operator can configure native Kubernetes and can manage and operate Prometheus and Alertmanager clusters.

The Operator introduces the following Kubernetes custom resource definitions (CRDs): Prometheus, ServiceMonitor, PrometheusRule, and Alertmanager. If you want to know more, you can visit the link:

Https://github.com/coreos/prometheus-operator/blob/master/Documentation/design.md

In our demonstration, we will use PrometheusRule to define custom rules.

First, we need to install Prometheus Operator using stable/prometheus-operator Helm chart, download the link:

Https://github.com/helm/charts/tree/master/stable/prometheus-operator

The default installer will deploy the following components: prometheus-operator, prometheus, alertmanager, node-exporter, kube-state-metrics, and grafana. By default, Prometheus will grab the main components of Kubernetes: kube-apiserver, kube-controller-manager, and etcd.

Install Prometheus softwar

Preparation in advance

To successfully execute this demo, you need to prepare the following:

A Google Cloud Platform account (free package is fine). Any other cloud is fine.

Rancher v2.3.5 (the latest version when the article was published)

Kubernetes cluster running on GKE (version 1.15.9-gke.12.) (using EKS or AKS is also possible)

Install Helm binary on your computer

Start an instance of Rancher

Just follow this intuitive getting started guide:

Https://rancher.com/quick-start

Deploy a GKE cluster using Rancher

Use Rancher to set up and configure your Kubernetes cluster:

Https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/gke/

After the deployment is complete, and the appropriate credential and endpoint information are configured for the kubeconfig file, you can use kubectl to point to that particular cluster.

Deploy Prometheus softwar

First, check the version of Helm we are running

$helm versionversion.BuildInfo {Version: "v3.1.2", GitCommit: "d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState: "clean", GoVersion: "go1.13.8"}

When we use Helm 3, we need to add a stable image repository because it is not set up by default.

$helm repo add stable https://kubernetes-charts.storage.googleapis.com"stable" has been added to your repositories$ helm repo updateHang tight while we grab the latest from your chart repositories.Successfully got an update from the "stable" chart repositoryUpdate Complete. ⎈ Happy Helming! ⎈ $helm repo listNAME URLstable https://kubernetes-charts.storage.googleapis.com

After the Helm configuration is complete, we can begin to install prometheus-operator

$kubectl create namespace monitoringnamespace/monitoring created$ helm install-- namespace monitoring demo stable/prometheus-operatormanifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest_sorter.go:192: info: skipping unknown hook: "crd-install" manifest _ sorter.go:192: info: skipping unknown hook: "crd-install" NAME: demoLAST DEPLOYED: Sat Mar 14 09:40:35 2020NAMESPACE: monitoringSTATUS: deployedREVISION: 1NOTES:The Prometheus Operator has been installed. Check its status by running: kubectl-- namespace monitoring get pods-l "release=demo" Visit https://github.com/coreos/prometheus-operator for instructions on howto create & configure Alertmanager and Prometheus instances using the Operator. Rules

In addition to monitoring, Prometheus also lets us create rules that trigger alarms. These rules are based on Prometheus's expression language. As long as the condition is met, the alarm is triggered and sent to Alertmanager. After that, we will see the specific form of the rules.

Let's go back to demo. After the Helm has been deployed, we can examine what pod has been created:

$kubectl-n monitoring get podsNAME READY STATUS RESTARTS AGEalertmanager-demo-prometheus-operator-alertmanager-0 2 9l57b 2 Running 0 61sdemo-grafana-5576fbf669-9l57b 3 Running 0 72sdemo-kube-state-metrics-67bf64b7f4-4786k 1 Running 0 72sdemo-prometheus-node-exporter-ll8zx 1 to 1 Running 0 72sdemo-prometheus-node-exporter-nqnr6 1 to 1 Running 0 72sdemo-prometheus-node-exporter-sdndf 1 to 1 Running 0 72sdemo-prometheus-operator-operator-b9c9b5457-db9dj 2 to 2 Running 0 72 sprometheus- Demo-prometheus-operator-prometheus-0 3/3 Running 1 50s

In order to access Prometheus and Alertmanager from a web browser, we need to use port forwarding.

Since the GCP instance is used in this example, and all kubectl commands are run from that instance, we use the external IP address of the instance to access the resource.

$kubectl port-forward-- address 0.0.0.0-n monitoring prometheus-demo-prometheus-operator-prometheus-0 9090 > / dev/null 2 > & 1 & $kubectl port-forward-- address 0.0.0.0-n monitoring alertmanager-demo-prometheus-operator-alertmanager-0 9093 > / dev/null 2 > & 1 &

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.