Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to deploy Prometheus federation manually in Kubernetes". In daily operation, I believe many people have doubts about how to manually deploy Prometheus federation in Kubernetes. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "how to deploy Prometheus federation manually in Kubernetes". Next, please follow the editor to study!

When we have multiple Kubernetes clusters, we need to summarize the metrics. As shown in the figure above, we assume that we deploy a Federate of Prometheus externally, and then collect the kube-system and default namespace in the current K8s.

Environment

My local environment uses one-click deployment of sealos, mainly for ease of testing.

OSKubernetesHostNameIPServiceUbuntu 18.041.17.7sealos-k8s-m1192.168.1.151node-exporter prometheus-federate-0Ubuntu 18.041.17.7sealos-k8s-m2192.168.1.152node-exporter grafana alertmanager-0Ubuntu 18.041.17.7sealos-k8s-m3192.168.1.150node-exporter alertmanager-1Ubuntu 18.041.17.7sealos-k8s-node1192.168.1.153node-exporter prometheus-0 kube-state-metricsUbuntu 18.041.17.7sealos-k8s-node2192.168.1.154node-exporter prometheus-1Ubuntu 18 .041.17.7sealos-k8s-node2192.168.1.155node-exporter prometheus-2 deployment Prometheus federation cluster

Create an prometheus-federate data directory

# execute on M1

Mkdir / data/prometheus-federate/

Chown-R 65534R 65534 / data/prometheus-federate/

Create a Prometheus federated StorageClass profile

Cd / data/manual-deploy/prometheus/

Cat prometheus-federate-storageclass.yaml

ApiVersion: storage.k8s.io/v1

Kind: StorageClass

Metadata:

Name: prometheus-federate-lpv

Provisioner: kubernetes.io/no-provisioner

VolumeBindingMode: WaitForFirstConsumer

Create a Prometheus federated pv profile

ApiVersion: v1

Kind: PersistentVolume

Metadata:

Name: prometheus-federate-lpv-0

Spec:

Capacity:

Storage: 10Gi

VolumeMode: Filesystem

AccessModes:

-ReadWriteOnce

PersistentVolumeReclaimPolicy: Retain

StorageClassName: prometheus-federate-lpv

Local:

Path: / data/prometheus-federate

NodeAffinity:

Required:

NodeSelectorTerms:

-matchExpressions:

-key: kubernetes.io/hostname

Operator: In

Values:

-sealos-k8s-m1

Create a Prometheus federated configmap profile

Cat prometheus-federate-configmap.yaml

ApiVersion: v1

Kind: ConfigMap

Metadata:

Name: prometheus-federate-config

Namespace: kube-system

Data:

Alertmanager_rules.yaml: |

Groups:

-name: example

Rules:

-alert: InstanceDown

Expr: up = = 0

For: 1m

Labels:

Severity: page

Annotations:

Summary: "Instance {{$labels.instance}} down"

Description: "{{$labels.instance}} of job {{$labels.job}} has been down for more than 1 minutes."

-alert: NodeMemoryUsage

Expr: (node_memory_MemTotal_bytes-(node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes)) / node_memory_MemTotal_bytes * 100 > 80

For: 1m

Labels:

Team: ops

Annotations:

Summary: "cluster: {{$labels.cluster}} {{$labels.instance}: High Memory usage detected"

Description: "{{$labels.instance}}: Memory usage is above 55% (current value is: {{$value}}"

Prometheus.yml: |

Global:

Scrape_interval: 30s

Evaluation_interval: 30s

Alerting:

Alertmanagers:

-static_configs:

-targets:

-alertmanager-0.alertmanager-operated:9093

-alertmanager-1.alertmanager-operated:9093

Rule_files:

-"/ etc/prometheus/alertmanager_rules.yaml"

Scrape_configs:

-job_name: 'federate'

Scrape_interval: 30s

Honor_labels: true

Metrics_path:'/ federate'

Params:

'match []':

-'{job=~ "kubernetes.*"}'

-'{job= "prometheus"}'

Static_configs:

-targets:

-'prometheus-0.prometheus:9090'

-'prometheus-1.prometheus:9090'

-'prometheus-2.prometheus:9090'

Create a statefulse file for the Prometheus federation

Cat prometheus-federate-statefulset.yaml

ApiVersion: apps/v1

Kind: StatefulSet

Metadata:

Name: prometheus-federate

Namespace: kube-system

Labels:

K8s-app: prometheus-federate

Kubernetes.io/cluster-service: "true"

Spec:

ServiceName: "prometheus-federate"

PodManagementPolicy: "Parallel"

Replicas: 1

Selector:

MatchLabels:

K8s-app: prometheus-federate

Template:

Metadata:

Labels:

K8s-app: prometheus-federate

Annotations:

Scheduler.alpha.kubernetes.io/critical-pod:''

Spec:

Affinity:

PodAntiAffinity:

RequiredDuringSchedulingIgnoredDuringExecution:

-labelSelector:

MatchExpressions:

-key: k8s-app

Operator: In

Values:

-prometheus-federate

TopologyKey: "kubernetes.io/hostname"

PriorityClassName: system-cluster-critical

HostNetwork: true

DnsPolicy: ClusterFirstWithHostNet

Containers:

-name: prometheus-federate-configmap-reload

Image: "jimmidyson/configmap-reload:v0.4.0"

ImagePullPolicy: "IfNotPresent"

Args:

-volume-dir=/etc/config

-webhook-url= http://localhost:9091/-/reload

VolumeMounts:

-name: config-volume

MountPath: / etc/config

ReadOnly: true

Resources:

Limits:

Cpu: 10m

Memory: 10Mi

Requests:

Cpu: 10m

Memory: 10Mi

SecurityContext:

RunAsUser: 0

Privileged: true

-image: prom/prometheus:v2.20.0

ImagePullPolicy: IfNotPresent

Name: prometheus

Command:

-"/ bin/prometheus"

Args:

"--web.listen-address=0.0.0.0:9091"

"--config.file=/etc/prometheus/prometheus.yml"

"--storage.tsdb.path=/prometheus"

"--storage.tsdb.retention=24h"

"--web.console.libraries=/etc/prometheus/console_libraries"

"--web.console.templates=/etc/prometheus/consoles"

"--web.enable-lifecycle"

Ports:

-containerPort: 9091

Protocol: TCP

VolumeMounts:

-mountPath: "/ prometheus"

Name: prometheus-federate-data

-mountPath: "/ etc/prometheus"

Name: config-volume

ReadinessProbe:

HttpGet:

Path: /-/ ready

Port: 9091

InitialDelaySeconds: 30

TimeoutSeconds: 30

LivenessProbe:

HttpGet:

Path: /-/ healthy

Port: 9091

InitialDelaySeconds: 30

TimeoutSeconds: 30

Resources:

Requests:

Cpu: 100m

Memory: 100Mi

Limits:

Cpu: 1000m

Memory: 2500Mi

SecurityContext:

RunAsUser: 0

Privileged: true

ServiceAccountName: prometheus

Volumes:

-name: config-volume

ConfigMap:

Name: prometheus-federate-config

VolumeClaimTemplates:

-metadata:

Name: prometheus-federate-data

Spec:

AccessModes: ["ReadWriteOnce"]

StorageClassName: "prometheus-federate-lpv"

Resources:

Requests:

Storage: 5Gi

Create a svc file for the Prometheus federation

Cat prometheus-service-statefulset.yaml

ApiVersion: v1

Kind: Service

Metadata:

Name: prometheus

Namespace: kube-system

Spec:

Ports:

-name: prometheus

Port: 9090

TargetPort: 9090

Selector:

K8s-app: prometheus

ClusterIP: None

Deployment

Cd / data/manual-deploy/prometheus/

Prometheus-federate-configmap.yaml

Prometheus-federate-pv.yaml

Prometheus-federate-service-statefulset.yaml

Prometheus-federate-statefulset.yaml

Prometheus-federate-storageclass.yaml

Kubectl apply-f prometheus-federate-storageclass.yaml

Kubectl apply-f prometheus-federate-pv.yaml

Kubectl apply-f prometheus-federate-configmap.yaml

Kubectl apply-f prometheus-federate-statefulset.yaml

Kubectl apply-f prometheus-federate-service-statefulset.yaml

Verification

# pv

Kubectl-n kube-system get pvc | grep federate

Prometheus-federate-data-prometheus-federate-0 Bound prometheus-federate-lpv-0 10Gi RWO prometheus-federate-lpv 4h

Kubectl-n kube-system get pod | grep federate

Prometheus-federate-0 2/2 Running 0 2d4h

In this regard, the federation configuration is complete. You can visit 192.168.1.151 targets 9091 in the browser to view the corresponding rules information and the configured rules rules, and trigger the alarm to see that the Alertmanager cluster has been successfully deployed.

At this point, the study on "how to manually deploy the Prometheus federation in Kubernetes" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.