Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to use Prometheus". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to use Prometheus.

Preparation in advance

A Google Cloud platform account (free), and other clouds are the same.

Rancher v2.4.2 (the latest version at the time of publication)

Kubernetes cluster running on GKE (version 1.15.11-gke.3) (EKS or AKS is also available)

Start an instance of Rancher

First, start an instance of Rancher. You can follow the instructions of Rancher to start:

Https://www.rancher.cn/quick-start/

Deploy a GKE cluster using Rancher

Use Rancher to set up and configure a Kubernetes cluster. You can visit the link below to get the documentation:

Https://rancher2.docs.rancher.cn/docs/cluster-provisioning/_index

Deploy Prometheus

We will use Rancher's app store to install Prometheus. Rancher's App Store mainly collects a lot of Helm Chart so that users can deploy applications repeatedly.

After our cluster is up and running, let's select the default project created for it under the "Apps" tab, and then click the "Launch" button.

Now let's search for the chart we are interested in. We can set a lot of fields-- but for this demo we will keep the default values. You can find useful information about these values in the Detailed Description section. There is no need to worry about problems, just check their use. At the bottom of the page, click [Launch]. Prometheus Server and Alertmanager will be installed and configured.

When the installation is complete, the page looks like this:

Next, we need to create a Services to access Prometheus Server as well as Alertmanager. Click the workload tab under the resources. In the load balancing section, we can see that it is not configured yet. Click Import YAML, select prometheus namespace, copy two YAML at once, and click Import. Later you will see how we know to use those specific ports and components tag.

ApiVersion: v1kind: Servicemetadata: name: prometheus-servicespec: type: LoadBalancer ports:-port: 80 targetPort: 9090 protocol: TCP selector: component: serverapiVersion: v1kind: Servicemetadata: name: alertmanager-servicespec: type: LoadBalancer ports:-port: 80 targetPort: 9093 protocol: TCP selector: component: alertmanager

When you are finished, service will display Active.

In the drop-down menu with the vertical ellipsis on the right, you can find IP and click View/Edit YAML. At the bottom of the yaml file, you will see something similar:

Status: loadBalancer: ingress:-ip: 34.76.22.14

Visiting IP will show us the GUI of Prometheus Server and Alertmanager. You will find that there is nothing to view at this time, because the rules have not been defined and alerts have not been configured.

Add Rul

The rules allow us to trigger the alarm. These rules are based on the expression language of Prometheus. Whenever the conditions are met, the alarm is triggered and sent to the Alertmanager.

Now let's see how we can add rules.

Under the resources-> workload tab, we can see what Deployment creates when running chart. Let's take a closer look at prometheus-server and prometheus-alertmanager.

We start with the first and understand its configuration, how we edit it and know which port the service is running on. Click the vertical ellipsis menu button and click View/Edit YAML.

First, we see two containers associated with Deplolyment: prometheus-server-configmap-reload and prometheus-server. The exclusive section of the container prometheus-server has some information about it:

As we know, Prometheus is configured through prometheus.yml. This file (and other files listed in serverFiles) will be mounted to server pod. In order to add / edit rules, we need to modify this file. In fact, this is a Config Map, which can be found under the tab of Resources Config. Click the vertical omit menu button and Edit. In the rules section, let's add a new rule and click Save.

Groups:-name: memory demo alert rules:-alert: High Pod Memory expr: container_memory_usage_bytes {pod_name=~ "nginx-.*", imageframes = "" Containercontainers = "POD"} > 5000000 for: 1m labels: severity: critical annotations: summary: High Memory Usage-name: cpu demo alert rules:-alert: High Pod CPU expr: rate (container_cpu_usage_seconds_total {pod_name=~ "nginx-.*", imageboxes = "" Containerships = "POD"} [5m]) > 0.04 for: 1m labels: severity: critical annotations: summary: High CPU Usage

The rules will be loaded automatically by Prometheus Server, and then we can see them in Prometheus Server GUI:

This is the interpretation of the above two rules:

Container_memory_usage_bytes: current memory usage (in bytes), including all memory, whenever accessed.

Container_cpu_usage_seconds_total: cumulative CPU time (in seconds)

All metrics can be found on the following pages:

Https://github.com/google/cadvisor/blob/master/metrics/prometheus.go

All regular expressions in Prometheus use RE2 syntax. With regular expressions, we can only select time series for Pod whose names match a specific pattern. In our example, we need to look for a pod that starts with nginx- and exclude "POD" because this is the parent cgroup of the container and displays statistics for all containers within the pod.

For container_cpu_usage_seconds_total, we use what is called a Subquery. It will return to our target every 5 minutes.

If you want to know more about queries and examples, you can check out the official Prometheus documentation.

Configure alarm

Whenever there is a problem, the alarm can immediately remind us, so that we can immediately know that an error has occurred in the system. Prometheus provides alerts through the Alertmanager component.

The procedure is the same as that of Prometheus Server: under the Resources-> workload tab, click the button on the right menu bar of prometheus-alertmanager, select View/Edit YAML, and check its configuration:

Alertmanager is configured through alertmanager.yml. This file (and other files listed in alertmanagerFiles) will be mounted on alertmanager pod. Next we need to modify the configMap associated with alertmanager to make it easier to set alarms. Under the Config tab, click the menu bar of the prometheus-alertmanager line, and then select Edit. Use the following code instead of the basic configuration:

Global: resolve_timeout: 5mroute: group_by: [Alertname] # Send all notifications to me. Receiver: demo-alert group_wait: 30s group_interval: 5m repeat_interval: 12h routes:-match: alertname: DemoAlertName receiver: "demo-alert" receivers:-name: demo-alert email_configs:-to: your_email@gmail.com from: from_email@gmail.com # Your smtp server address smarthost: smtp.gmail.com:587 auth_username: from_email@gmail.com Auth_identity: from_email@gmail.com auth_password: 16letter_generated token # you can use gmail account password But better create a dedicated token for this headers: From: from_email@gmail.com Subject: "Demo ALERT"

The new configuration will be reloaded by Alertmanager, and we can see GUI under the Status tab.

Test the End-to-End scenario

Let's deploy some components for monitoring. Deploying a simple nginx deployment is sufficient for practice. Using Rancher GUI, under the Resources-> workload tab, click Import YAML, paste the following code (this time using the default namespace), and click Import:

ApiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2kind: Deploymentmetadata: name: nginx-deploymentspec: selector: matchLabels: app: nginx replicas: 3 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx spec: containers:-name: nginx image: nginx:1.7.9 ports:-containerPort: 80

In Prometheus UI, we looked at some metrics to use one of the two expressions previously configured for alerts:

Rate (container_cpu_usage_seconds_total {pod_name=~ "nginx-.*", imageboxes = "", containerships = "POD"} [5m])

Let's add some payload to one of the Pod to see the change in value. When the value is greater than 0.04, we should get an alarm. To do this, we need to select one of the nginx Deployment Pod and click Execute Shell. In which we will execute a command:

There are three stages of alarm:

Inactive- condition is not satisfied

Pending- satisfies the condition

Firing- alarm is triggered

We have seen that the alarm is in the inactive state, so continue to increase the load on the CPU so that we can observe the remaining two states:

As soon as the alarm is triggered, it will be displayed in Alertmanager:

Configure Alertmanager to send email when we receive an alarm. If we look at our inbox, we will see something similar:

Thank you for your reading, the above is the content of "how to use Prometheus", after the study of this article, I believe you have a deeper understanding of how to use Prometheus, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.