Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to use Prometheus and Thanos for high-availability K8S monitoring. The content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Introduce the necessity of high availability of Prometheus

In the past few years, the use of Kubernetes has increased several times. Obviously, Kubernetes is the best choice for container choreography. At the same time, Prometheus is also considered an excellent choice for monitoring containerized and non-containerized workloads. Monitoring is an important concern for any infrastructure, and we should ensure that our monitoring settings are highly available and scalable to meet the growing infrastructure needs, especially with Kubernetes.

Therefore, today we will deploy a clustered Prometheus setup that not only responds flexibly to node failures, but also ensures proper data archiving for future reference. Our setup is also so scalable that we can span multiple Kubernetes clusters under the same monitoring umbrella.

Current plan

Most Prometheus deployments use pod with persistent volumes, while Prometheus is extended using federated mechanisms. But not all data can be aggregated using federated mechanisms, where you often need a mechanism to manage Prometheus configuration when you add additional servers.

Solution method

Thanos aims to solve the above problems. With the help of Thanos, we can not only copy multiple instances of Prometheus and deduplicate data between them, but also archive the data to long-term storage such as GCS or S3.

Implementation process Thanos architecture

Photo Source: https://thanos.io/quick-tutorial.md/

Thanos consists of the following components:

Thanos sidecar: this is the main component that runs on Prometheus. It reads and archives data on the object store. In addition, it manages the configuration and lifecycle of Prometheus. To distinguish each Prometheus instance, the sidecar component injects external tags into the Prometheus configuration. This component can run queries on the PromQL interface of the Prometheus server. The Sidecar component can also listen to the Thanos gRPC protocol and translate queries between gRPC and REST.

Thanos storage: this component implements Store API on top of the historical data in the object storage bucket, acting primarily as an API gateway, so it does not require a large amount of local disk space. It joins a Thanos cluster at startup and publishes the data it can access. It keeps a small amount of information about all remote chunks on the local disk and keeps it synchronized with bucket. Typically, this data can be safely deleted on reboot, but it increases startup time.

Thanos queries: the query component listens on HTTP and translates queries into Thanos gRPC format. It summarizes query results from different sources and reads data from Sidecar and Store. In the HA setting, it even deduplicates the query results.

Run-time deduplication of HA groups

Prometheus is stateful and replication of its database is not allowed. This means that improving high availability by running multiple copies of Prometheus is not easy to use. Simple load balancing doesn't work, for example, a copy may start after some crash, but querying such a copy will cause it to have a small gap during shutdown. You have a second copy that may be starting, but it may shut down at another time (such as a rolling restart), so the load balancing on these replicas will not work properly.

Thanos Querier extracts data from two copies and deduplicates these signals, thus filling the gap (gap) for Querier users.

The Thanos Compact component applies the compactor of the Prometheus 2.0 storage engine to the block data store in the object store. It is usually not semantic concurrency security and must be deployed singleton for bucket. It is also responsible for data downsampling-5m downsampling after 40 hours and 1h downsampling after 10 days.

Thanos Ruler basically has the same effect as Prometheus's rules, except that it can communicate with Thanos components.

Preliminary preparation for allocation

To fully understand this tutorial, you need to prepare the following:

Have some knowledge of Kubernetes and using kubectl.

The running Kubernetes cluster has at least 3 nodes (in this demo, the GKE cluster is used)

Implement the Ingress Controller and Ingress objects (Nginx Ingress Controller is used in this demo). Although this is not mandatory, it is strongly recommended to reduce the number of external endpoints created.

Create credentials for the Thanos component to access the object store (in this case, GCS bucket).

Create 2 GCS bucket and name them Prometheus-long-term and thanos-ruler.

Create a service account with the role of Storage Object Admin.

Download the key file as a json certificate and name it thanos-gcs-credentials.json.

Create a Kubernetes sercret with credentials

Kubectl create secret generic thanos-gcs-credentials-from-file=thanos-gcs-credentials.json

Deploy various components

Deploy Prometheus service accounts, Clusterroler, and Clusterrolebinding

ApiVersion: v1kind: Namespacemetadata: name: monitoring---apiVersion: v1kind: ServiceAccountmetadata: name: monitoring namespace: rbac.authorization.k8s.io/v1beta1kind: ClusterRolemetadata: name: monitoring namespace: monitoringrules:- apiGroups: ["] resources:-nodes-nodes/proxy-services-endpoints-pods verbs: [" get "," list " "watch"]-apiGroups: ["] resources:-configmaps verbs: [" get "]-nonResourceURLs: [" / metrics "] verbs: [" get "]-apiVersion: rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata: name: monitoringsubjects:-kind: ServiceAccount name: monitoring namespace: monitoringroleRef: kind: ClusterRole Name: monitoring apiGroup: rbac.authorization.k8s.io

The above manifest creates the monitoring namespace as well as the service account, clusterrole, and clusterrolebinding required by Prometheus.

Deploy Prometheues configuration configmapapiVersion: v1kind: ConfigMapmetadata: name: prometheus-server-conf labels: name: prometheus-server-conf namespace: monitoringdata: prometheus.yaml.tmpl: |-global: scrape_interval: 5s evaluation_interval: 5s external_labels: cluster: prometheus-ha # Each Prometheus has to have unique labels. Replica: $(POD_NAME) rule_files:-/ etc/prometheus/rules/*rules.yaml alerting: # We want our alerts to be deduplicated # from different replicas. Alert_relabel_configs:-regex: replica action: labeldrop alertmanagers:-scheme: http path_prefix: / static_configs:-targets: ['alertmanager:9093'] scrape_configs:-job_name: kubernetes-nodes-cadvisor scrape_interval: 10s scrape_timeout: 10s scheme: https tls_config: ca_file: / Var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: / var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs:-role: node relabel_configs:-action: labelmap regex: _ _ meta_kubernetes_node_label_ (. +) # Only for Kubernetes ^ 1.7.3. # See: https://github.com/prometheus/prometheus/issues/2916-target_label: _ _ address__ replacement: kubernetes.default.svc:443-source_labels: [_ _ meta_kubernetes_node_name] regex: (. +) target_label: _ _ metrics_path__ replacement: / api/v1/nodes/$ {1} / proxy/metrics / cadvisor metric_relabel_configs:-action: replace source_labels: [id] regex:'^ / machine\ .slice/machine-rkt\\ x2d ([^\] +)\ .service $'target_label: rkt_container_name replacement:' ${2}-${1}'- action: replace Source_labels: [id] regex:'^ / system\ .slice / (. +)\ .service $'target_label: systemd_service_name replacement:' ${1}'- job_name: 'kubernetes-pods' kubernetes_sd_configs:-role: pod relabel_configs:-action: labelmap regex: _ _ meta_kubernetes_pod_label_ (. +)-source_labels: [_ meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace-source_labels: [_ _ meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name-source_labels: [_ _ meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep Regex: true-source_labels: [_ meta_kubernetes_pod_annotation_prometheus_io_scheme] action: replace target_label: _ _ scheme__ regex: (https?)-source_labels: [_ _ meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: _ _ metrics_path__ Regex: (. +)-source_labels: [_ _ address__ _ _ meta_kubernetes_pod_prometheus_io_port] action: replace target_label: _ _ address__ regex: ([^:] +) (?::\ d +)? (\ d +) replacement: $1role 2-job_name: 'kubernetes-apiservers' kubernetes_sd_configs:-role: endpoints scheme: https tls_config: ca_file: / var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: / var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs:-source_labels: [_ _ meta_kubernetes_namespace _ _ meta_kubernetes_service_name, _ _ meta_kubernetes_endpoint_port_name] action: keep regex: default Kubernetes Https-job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs:-role: endpoints relabel_configs:-action: labelmap regex: _ meta_kubernetes_service_label_ (. +)-source_labels: [_ meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace-source_labels: [_ meta _ kubernetes_service_name] action: replace target_label: kubernetes_name-source_labels: [_ _ meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true-source_labels: [_ meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: _ _ scheme__ Regex: (https?)-source_labels: [_ _ meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: _ _ metrics_path__ regex: (. +)-source_labels: [_ _ address__ _ _ meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: _ _ address__ regex: (. +) (?::\ d +) (\ d +) replacement: $1RV 2

The above Configmap creates the Prometheus profile template. This profile template will be read by the Thanos sidecar component, which will generate the actual configuration file, which in turn will be consumed by Prometheus containers running in the same pod. It is extremely important to add the external_labels section to the configuration file so that Querier can delete data repeatedly based on this.

Deploy Prometheus Rules configmap

This will create our alarm rules, which will be forwarded to alertmanager for delivery.

ApiVersion: v1kind: ConfigMapmetadata: name: prometheus-rules labels: name: prometheus-rules namespace: monitoringdata: alert-rules.yaml: |-groups:-name: Deployment rules:-alert: Deployment at 0 Replicas annotations: summary: Deployment {{$labels.deployment}} in {{$labels.namespace} is currently having no pods running expr: | sum (kube_deployment_status_replicas {pod_template_hash= "}) by (deployment Namespace)

< 1 for: 1m labels: team: devops - alert: HPA Scaling Limited annotations: summary: HPA named {{$labels.hpa}} in {{$labels.namespace}} namespace has reached scaling limited state expr: | (sum(kube_hpa_status_condition{condition="ScalingLimited",status="true"}) by (hpa,namespace)) == 1 for: 1m labels: team: devops - alert: HPA at MaxCapacity annotations: summary: HPA named {{$labels.hpa}} in {{$labels.namespace}} namespace is running at Max Capacity expr: | ((sum(kube_hpa_spec_max_replicas) by (hpa,namespace)) - (sum(kube_hpa_status_current_replicas) by (hpa,namespace))) == 0 for: 1m labels: team: devops - name: Pods rules: - alert: Container restarted annotations: summary: Container named {{$labels.container}} in {{$labels.pod}} in {{$labels.namespace}} was restarted expr: | sum(increase(kube_pod_container_status_restarts_total{namespace!="kube-system",pod_template_hash=""}[1m])) by (pod,namespace,container) >

0 for: 0m labels: team: dev-alert: High Memory Usage of Container annotations: summary: Container named {{labels.container}} in {{$labels.pod}} in {{$labels.namespace}} is using more than 75% of Memory Limit expr: ((sum (container_memory_usage_bytes {imageboxes = "", containercontainer names = "POD") Namespaceworthy = "kube-system"}) by (namespace,container_name,pod_name) / sum (container_spec_memory_limit_bytes {imageframes = "", containercontainer names = "POD", namespaceworthy = "kube-system"}) by (namespace,container_name,pod_name)) * 100)

< +Inf ) >

75 for: 5m labels: team: dev-alert: High CPU Usage of Container annotations: summary: Container named {{labels.container}} in {{$labels.pod}} in {{$labels.namespace}} is using more than 75% of CPU Limit expr: | (sum (irate (container_cpu_usage_seconds_total {imageboxes = "", containercontainer names = "POD") Namespacegrounds = "kube-system"} [30s]) by (namespace,container_name,pod_name) / sum (container_spec_cpu_quota {imageframes = "", container names = "POD", namespacestones = "kube-system"} / container_spec_cpu_period {imageframes = "", container names = "POD", namespacestones = "kube-system"}) by (namespace,container_name) Pod_name)) * 100) > 75 for: 5m labels: team: dev-name: Nodes rules:-alert: High Node Memory Usage annotations: summary: Node {{$labels.kubernetes_io_hostname}} has more than 80 memory used. Plan Capcity expr: (sum (container_memory_working_set_bytes {id= "/") Containercontainer names = "POD"}) by (kubernetes_io_hostname) / sum (machine_memory_bytes {}) by (kubernetes_io_hostname) * 100) > 80 for: 5m labels: team: devops-alert: High Node CPU Usage annotations: summary: Node {{$labels.kubernetes_io_hostname}} has more than 80 allocatable cpu used. Plan Capacity. Expr: (sum (rate (container_cpu_usage_seconds_total {id= "/") Containercontainer names = "POD"} [1m]) by (kubernetes_io_hostname) / sum (machine_cpu_cores) by (kubernetes_io_hostname) * 80 for: 5m labels: team: devops-alert: High Node Disk Usage annotations: summary: Node {{$labels.kubernetes_io_hostname}} has more than 85% disk used. Plan Capacity. Expr: | (sum (container_fs_usage_bytes {device=~ "^ / dev/ [sv] d [Amurz] [1-9] $", id= "/", containerized nameplate = "POD"}) by (kubernetes_io_hostname) / sum (container_fs_limit_bytes {containerized nameplate = "POD", device=~ "^ / dev/ [sv] d [aMuz] [1-9] $" Id= "/"}) by (kubernetes_io_hostname) * 100 > 85 for: 5m labels: team: devops deployment Prometheus Stateful SetapiVersion: storage.k8s.io/v1beta1kind: StorageClassmetadata: name: fast namespace: monitoringprovisioner: kubernetes.io/gce-pdallowVolumeExpansion: true---apiVersion: apps/v1beta1kind: StatefulSetmetadata: name: prometheus namespace: replicas: 3 serviceName: prometheus-service template: metadata: labels: app: Prometheus thanos-store-api: "true" spec: serviceAccountName: monitoring containers:-name: prometheus image: prom/prometheus:v2.4.3 args:-"--config.file=/etc/prometheus-shared/prometheus.yaml"-- storage.tsdb.path=/prometheus/ "--web.enable-lifecycle" -"--storage.tsdb.no-lockfile"-- "--storage.tsdb.min-block-duration=2h"-- storage.tsdb.max-block-duration=2h "ports:-name: prometheus containerPort: 9090 volumeMounts:-name: prometheus-storage mountPath: / prometheus/ -name: prometheus-config-shared mountPath: / etc/prometheus-shared/-name: prometheus-rules mountPath: / etc/prometheus/rules-name: thanos image: quay.io/thanos/thanos:v0.8.0 args:-"sidecar"-"- log.level=debug"- "--tsdb.path=/prometheus"-- "--prometheus.url= http://127.0.0.1:9090" -"-- objstore.config= {type: GCS Config: {bucket: prometheus-long-term} "-"-- reloader.config-file=/etc/prometheus/prometheus.yaml.tmpl "- -"-- reloader.config-envsubst-file=/etc/prometheus-shared/prometheus.yaml "- -"-- reloader.rule-dir=/etc/prometheus/rules/ "env:-name: POD_NAME valueFrom : fieldRef: fieldPath: metadata.name-name: GOOGLE_APPLICATION_CREDENTIALS value: / etc/secret/thanos-gcs-credentials.json ports:-name: http-sidecar containerPort: 10902-name: grpc containerPort: 10901 livenessProbe: httpGet : port: 10902 path: /-/ healthy readinessProbe: httpGet: port: 10902 path: /-/ ready volumeMounts:-name: prometheus-storage mountPath: / prometheus- name: prometheus-config-shared mountPath: / etc/prometheus- Shared/-name: prometheus-config mountPath: / etc/prometheus-name: prometheus-rules mountPath: / etc/prometheus/rules-name: thanos-gcs-credentials mountPath: / etc/secret readOnly: false securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 volumes: -name: prometheus-config configMap: defaultMode: 420 name: prometheus-server-conf-name: prometheus-config-shared emptyDir: {}-name: prometheus-rules configMap: name: prometheus-rules-name: thanos-gcs-credentials secret: secretName: thanos-gcs-credentials volumeClaimTemplates:-metadata: Name: prometheus-storage namespace: monitoring spec: accessModes: ["ReadWriteOnce"] storageClassName: fast resources: requests: storage: 20Gi

With regard to the manifest provided above, it is important to understand the following:

Prometheus is deployed as a stateful set with three replicas, each of which dynamically provides its own persistence volume.

The Prometheus configuration is generated by the Thanos sidecar container using the template file we created above.

Thanos handles data compression, so we need to set-- storage.tsdb.min-block-duration=2h and-- storage.tsdb.max-block-duration=2h.

The Prometheus stateful set is marked thanos-store-api: true so that each pod is discovered by the headless service we create next. It is this headless service that will be used by Thanos Querier to query the data of all Prometheus instances. We also apply the same tags to the Thanos Store and Thanos Ruler components so that they are also discovered by Querier and can be used to query metrics.

The GCS bucket credentials path is provided using the GOOGLE_APPLICATION_CREDENTIALS environment variable, and the configuration file is mounted on it by the secret we created in preparation.

Deploy Prometheus service apiVersion: v1kind: Servicemetadata: name: prometheus-0-service annotations: prometheus.io/scrape: "true" prometheus.io/port: "9090" namespace: monitoring labels: name: prometheusspec: selector: statefulset.kubernetes.io/pod-name: prometheus-0 ports:-name: prometheus port: 8080 targetPort: prometheus---apiVersion: v1kind: Servicemetadata: name: prometheus-1-service annotations: prometheus.io/ Scrape: "true" prometheus.io/port: "9090" namespace: monitoring labels: name: selector: statefulset.kubernetes.io/pod-name: prometheus-1 ports:-name: prometheus port: 8080 targetPort: prometheus---apiVersion: v1kind: Servicemetadata: name: prometheus-2-service annotations: prometheus.io/scrape: "true" prometheus.io/port: "9090" namespace: monitoring labels: name : prometheusspec: selector: statefulset.kubernetes.io/pod-name: prometheus-2 ports:-name: prometheus port: 8080 targetPort: prometheus---#This service creates a srv record for querier to find about store-api'sapiVersion: v1kind: Servicemetadata: name: thanos-store-gateway namespace: monitoringspec: type: ClusterIP clusterIP: None ports:-name: grpc port: 10901 targetPort: grpc selector: thanos-store-api: "true"

In addition to the above methods, you can also click on this article to learn how to quickly deploy and configure Prometheus services on Rancher.

We created a different service for each Prometheus pod in stateful set, although this is not necessary. These services are created for debugging purposes only. The purpose of thanos-store-gateway headless service has been explained above. We will use an ingress object later to expose the Prometheus service.

Deploy Prometheus QuerierapiVersion: v1kind: Namespacemetadata: name: monitoring---apiVersion: apps/v1kind: Deploymentmetadata: name: thanos-querier namespace: monitoring labels: app: thanos-querierspec: replicas: 1 selector: matchLabels: app: thanos-querier template: metadata: labels: app: thanos-querierspec: containers:-name: thanos image: quay.io/thanos/thanos:v0.8.0 args: -query-log.level=debug-query.replica-label=replica-store=dnssrv+thanos-store-gateway:10901 ports:-name: http containerPort: 10902-name: grpc containerPort: 10901 livenessProbe: httpGet: port: http path: /-/ healthy readinessProbe: HttpGet: port: http path: /-/ ready---apiVersion: v1kind: Servicemetadata: labels: app: thanos-querier name: thanos-querier namespace: monitoringspec: ports:-port: 9090 protocol: TCP targetPort: http name: http selector: app: thanos-querier

This is one of the main elements of Thanos deployment. Please note the following points:

Container parameters-store=dnssrv+thanos-store-gateway:10901 helps to find components of all metric data that should be queried.

The thanos-querier service provides a Web interface to run PromQL queries. It can also choose to repeatedly delete data in different Prometheus clusters.

This is the end point (end point) where we provide Grafana as the data source for all dashboard.

Deploy the Thanos storage gateway apiVersion: v1kind: Namespacemetadata: name: monitoring---apiVersion: StatefulSetmetadata: name: thanos-store-gateway namespace: monitoring labels: app: thanos-store-gatewayspec: replicas: 1 selector: matchLabels: app: thanos-store-gateway serviceName: thanos-store-gateway template: metadata: labels: app: thanos-store-gateway thanos-store-api: "true" spec: Containers:-name: thanos image: quay.io/thanos/thanos:v0.8.0 args:-"store"-"- log.level=debug"-"--data-dir=/data"-- objstore.config= {type: GCS Config: {bucket: prometheus-long-term} "-"-- index-cache-size=500MB "--chunk-pool-size=500MB" env:-name: GOOGLE_APPLICATION_CREDENTIALS value: / etc/secret/thanos-gcs-credentials.json ports:-name: http containerPort: 10902-name: grpc ContainerPort: 10901 livenessProbe: httpGet: port: 10902 path: /-/ healthy readinessProbe: httpGet: port: 10902 path: /-/ ready volumeMounts:-name: thanos-gcs-credentials mountPath: / etc/secret ReadOnly: false volumes:-name: thanos-gcs-credentials secret: secretName: thanos-gcs-credentials

This creates a storage component that provides metrics to Querier from the object store.

Deploy Thanos RulerapiVersion: v1kind: Namespacemetadata: name: monitoring---apiVersion: v1kind: ConfigMapmetadata: name: thanos-ruler-rules namespace: monitoringdata: alert_down_services.rules.yaml: | groups:-name: metamonitoring rules:-alert: PrometheusReplicaDown annotations: message: Prometheus replica in cluster {{$labels.cluster}} has disappeared from Prometheus target discovery. Expr: | sum (up {cluster= "prometheus-ha", instance=~ ". *: 9090", job= "kubernetes-service-endpoints"}) by (job) Cluster) < 3 for: 15s labels: severity: critical---apiVersion: StatefulSetmetadata: labels: app: thanos-ruler name: thanos-ruler namespace: monitoringspec: replicas: 1 selector: matchLabels: app: thanos-ruler serviceName: thanos-ruler template: metadata: labels: app: thanos-ruler thanos-store-api: "true" spec: containers: -name: thanos image: quay.io/thanos/thanos:v0.8.0 args:-rule- log.level=debug-data-dir=/data-eval-interval=15s-rule-file=/etc/thanos-ruler/*.rules.yaml-alertmanagers.url=http:/ / alertmanager:9093-query=thanos-querier:9090-"- objstore.config= {type: GCS Config: {bucket: thanos-ruler} "- label=ruler_cluster=" prometheus-ha "- label=replica=" $(POD_NAME) "env:-name: GOOGLE_APPLICATION_CREDENTIALS value: / etc/secret/thanos-gcs-credentials.json-name: POD_NAME valueFrom: fieldRef : fieldPath: metadata.name ports:-name: http containerPort: 10902-name: grpc containerPort: 10901 livenessProbe: httpGet: port: http path: /-/ healthy readinessProbe: httpGet: port: http Path: /-/ ready volumeMounts:-mountPath: / etc/thanos-ruler name: config-name: thanos-gcs-credentials mountPath: / etc/secret readOnly: false volumes:-configMap: name: thanos-ruler-rules name: config-name: thanos- Gcs-credentials secret: secretName: thanos-gcs-credentials---apiVersion: v1kind: Servicemetadata: labels: app: thanos-ruler name: thanos-ruler namespace: monitoringspec: ports:-port: 9090 protocol: TCP targetPort: http name: http selector: app: thanos-ruler

Now, if you start interactive shell in the same namespace as our workload and try to see which pods our thanos-store-gateway resolves to, you will see the following:

Root@my-shell-95cb5df57-4q6w8:/# nslookup thanos-store-gatewayServer: 10.63.240.10Address: 10.63.240.10#53Name: thanos-store-gateway.monitoring.svc.cluster.localAddress: 10.60.25.2Name: thanos-store-gateway.monitoring.svc.cluster.localAddress: 10.60.25.4Name: thanos-store-gateway.monitoring.svc.cluster.localAddress: 10.60.30.2Name: thanos-store-gateway.monitoring.svc. Cluster.localAddress: 10.60.30.8Name: thanos-store-gateway.monitoring.svc.cluster.localAddress: 10.60.31.2root@my-shell-95cb5df57-4q6w8:/# exit

The IP returned above corresponds to our Prometheus Pod, thanos-store, and thanos-ruler. This can be verified as:

$kubectl get pods-o wide-l thanos-store-api= "true" NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESprometheus-0 2 Running 0 100m 10.60.31.2 gke-demo-1-pool-1-649cbe02-jdnv prometheus-1 2 Running 0 14h 10.60.30.2 gke-demo-1-pool-1-7533d618-kxkd prometheus-2 2 Running 0 31h 10.60.25.2 gke-demo-1-pool-1-4e9889dd-27gc thanos-ruler-0 1 Running 0 100m 10.60.30 .8 gke-demo-1-pool-1-7533d618-kxkd thanos-store-gateway-0 1 Running 0 14h 10.60.25.4 gke-demo-1-pool-1-4e9889dd-27gc deployment AlertmanagerapiVersion: v1kind: Namespacemetadata: name: monitoring---kind: ConfigMapapiVersion: v1metadata: name: alertmanager namespace: monitoringdata: config.yml: |-global: resolve_timeout: 5m Slack_api_url: "" victorops_api_url: "" templates: -'/ etc/alertmanager-templates/*.tmpl' route: group_by: ['alertname' 'cluster' 'service'] group_wait: 10s group_interval: 1m repeat_interval: 5m receiver: default routes:-match: team: devops receiver: devops continue: true-match: team: dev receiver: dev continue: true receivers:-name:' default'-name: 'devops' victorops_configs: -api_key:''routing_key:' devops' message_type: 'CRITICAL' entity_display_name:' {{.CommonLabels.alertname}} 'state_message:' Alert: {{.CommonLabels.alertname}}. Summary: {{.CommonAnnotations.summary}}. RawData: {{.CommonLabels}} 'slack_configs:-channel:' # K8 send_resolved: true-name: 'dev' victorops_configs:-api_key:' 'routing_key:' dev' message_type: 'CRITICAL' entity_display_name: {{.CommonLabels.alertname}}' state_message : 'Alert: {{.CommonLabels.alertname}}. Summary: {{.CommonAnnotations.summary}}. RawData: {{.CommonLabels}} 'slack_configs:-channel:' # K8 send_resolved: true---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: alertmanager namespace: monitoringspec: replicas: 1 selector: matchLabels: app: alertmanager template: metadata: name: alertmanager labels: app: alertmanager spec: containers:-name: alertmanager image: prom/ Alertmanager:v0.15.3 args: -'--config.file=/etc/alertmanager/config.yml' -'- storage.path=/alertmanager' ports:-name: alertmanager containerPort: 9093 volumeMounts:-name: config-volume mountPath: / etc/alertmanager-name: alertmanager mountPath: / alertmanager volumes: -name: config-volume configMap: name: alertmanager-name: alertmanager emptyDir: {}-apiVersion: v1kind: Servicemetadata: annotations: prometheus.io/scrape: 'true' prometheus.io/path:' / metrics' labels: name: alertmanager name: alertmanager namespace: monitoringspec: selector: app: alertmanager ports:-name: alertmanager protocol: TCP port: 9093 targetPort: 9093

This will create our Alertmanager deployment, which will generate all alerts according to the Prometheus rules.

Deploy Kubestate metrics apiVersion: v1kind: Namespacemetadata: name: monitoring---apiVersion: rbac.authorization.k8s.io/v1# kubernetes versions before 1.8.0 should use rbac.authorization.k8s.io/v1beta1kind: ClusterRoleBindingmetadata: name: kube-state-metricsroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kube-state-metricssubjects:- kind: ServiceAccount name: kube-state-metrics namespace: monitoring---apiVersion: rbac.authorization.k8s.io/v1# kubernetes versions before 1.8.0 should Use rbac.authorization.k8s.io/v1beta1kind: ClusterRolemetadata: name: kube-state-metricsrules:- apiGroups: ["] resources:-configmaps-secrets-nodes-pods-services-resourcequotas-replicationcontrollers-limitranges-persistentvolumeclaims-persistentvolumes-namespaces-endpoints verbs: [" list " "watch"]-apiGroups: ["extensions"] resources:-daemonsets-deployments-replicasets verbs: ["list", "watch"]-apiGroups: ["apps"] resources:-statefulsets verbs: ["list", "watch"]-apiGroups: ["batch"] resources:-cronjobs-jobs verbs: ["list", "watch"]-apiGroups: ["autoscaling"] resources:-horizontalpodautoscalers verbs: ["list" "watch"]-apiVersion: rbac.authorization.k8s.io/v1# kubernetes versions before 1.8.0 should use rbac.authorization.k8s.io/v1beta1kind: RoleBindingmetadata: name: kube-state-metrics namespace: monitoringroleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kube-state-metrics-resizersubjects:- kind: ServiceAccount name: kube-state-metrics namespace: monitoring---apiVersion: rbac.authorization.k8s.io/v1# kubernetes versions before 1.8.0 should use rbac.authorization. K8s.io/v1beta1kind: Rolemetadata: namespace: monitoring name: kube-state-metrics-resizerrules:- apiGroups: [""] resources:-pods verbs: ["get"]-apiGroups: ["extensions"] resources:-deployments resourceNames: ["kube-state-metrics"] verbs: ["get" "update"]-apiVersion: v1kind: ServiceAccountmetadata: name: kube-state-metrics namespace: monitoring---apiVersion: apps/v1kind: Deploymentmetadata: name: kube-state-metrics namespace: monitoringspec: selector: matchLabels: k8s-app: kube-state-metrics replicas: 1 template: metadata: labels: k8s-app: kube-state-metrics spec: serviceAccountName: kube-state-metrics containers:-name: kube-state -metrics image: quay.io/mxinden/kube-state-metrics:v1.4.0-gzip.3 ports:-name: http-metrics containerPort: 8080-name: telemetry containerPort: 8081 readinessProbe: httpGet: / healthz port: 8080 initialDelaySeconds: 5 timeoutSeconds: 5-name: addon-resizer Image: k8s.gcr.io/addon-resizer:1.8.3 resources: limits: cpu: 150m memory: 50Mi requests: cpu: 150m memory: 50Mi env:-name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name -name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace command:-/ pod_nanny-container=kube-state-metrics-cpu=100m-extra-cpu=1m-memory=100Mi-extra-memory=2Mi- -- threshold=5-deployment=kube-state-metrics---apiVersion: v1kind: Servicemetadata: name: kube-state-metrics namespace: monitoring labels: k8s-app: prometheus.io/scrape: 'true'spec: ports:-name: http-metrics port: 8080 targetPort: http-metrics protocol: TCP-name: telemetry port: 8081 targetPort: telemetry protocol: TCP selector: k8s-app: kube-state-metrics

The deployment of Kubestate metrics requires forwarding some important container metrics that are not natively exposed by kubelet and therefore cannot be provided directly to Prometheus.

Deploy Node-Exporter DaemonsetapiVersion: v1kind: Namespacemetadata: name: monitoring---apiVersion: extensions/v1beta1kind: DaemonSetmetadata: name: node-exporter namespace: monitoring labels: name: node-exporterspec: template: metadata: labels: name: node-exporter annotations: prometheus.io/scrape: "true" prometheus.io/port: "9100" spec: hostPID: true hostIPC: true hostNetwork: true Containers:-name: node-exporter image: prom/node-exporter:v0.16.0 securityContext: privileged: true args:-- path.procfs=/host/proc-path.sysfs=/host/sys ports:-containerPort: 9100 protocol: TCP resources: Limits: cpu: 100m memory: 100Mi requests: cpu: 10m memory: 100Mi volumeMounts:-name: dev mountPath: / host/dev-name: proc mountPath: / host/proc-name: sys mountPath: / Host/sys-name: rootfs mountPath: / rootfs volumes:-name: proc hostPath: path: / proc-name: dev hostPath: path: / dev-name: sys hostPath: path: / sys-name: rootfs hostPath: path: /

Node-Exporter daemonset runs a node-exporter pod on each node and exposes very important node-related metrics that can be pulled by Prometheus instances.

Deploy GrafanaapiVersion: v1kind: Namespacemetadata: name: monitoring---apiVersion: storage.k8s.io/v1beta1kind: StorageClassmetadata: name: fast namespace: monitoringprovisioner: kubernetes.io/gce-pdallowVolumeExpansion: true---apiVersion: apps/v1beta1kind: StatefulSetmetadata: name: grafana namespace: monitoringspec: replicas: 1 serviceName: grafana template: metadata: labels: task: monitoring k8s-app: grafana spec: containers:-name: grafana image: k8s.gcr. Io/heapster-grafana-amd64:v5.0.4 ports:-containerPort: 3000 protocol: TCP volumeMounts:-mountPath: / etc/ssl/certs name: ca-certificates readOnly: true-mountPath: / var name: grafana-storage env:-name: GF_SERVER_HTTP_PORT value: "3000" # The following env variables are required to make Grafana accessible via # the kubernetes api-server proxy. On production clusters, we recommend # removing these env variables, setup auth for grafana, and expose the grafana # service using a LoadBalancer or a public IP. -name: GF_AUTH_BASIC_ENABLED value: "false"-name: GF_AUTH_ANONYMOUS_ENABLED value: "true"-name: GF_AUTH_ANONYMOUS_ORG_ROLE value: Admin-name: GF_SERVER_ROOT_URL # If you're only using the API Server proxy Set this value instead: # value: / api/v1/namespaces/kube-system/services/monitoring-grafana/proxy value: / volumes:-name: ca-certificates hostPath: path: / etc/ssl/certs volumeClaimTemplates:-metadata: name: grafana-storage namespace: monitoring spec: accessModes: ["ReadWriteOnce"] storageClassName: fast resources: Requests: storage: 5Gi---apiVersion: v1kind: Servicemetadata: labels: kubernetes.io/cluster-service: 'true' kubernetes.io/name: grafana name: grafana namespace: monitoringspec: ports:-port: 3000 targetPort: 3000 selector: k8s-app: grafana

This will create our Grafana deployment and service, which will be exposed using our Ingress object. To do this, we should add Thanos-Querier as the data source for our Grafana deployment:

Click to add data source

Set Name: DS_PROMETHEUS

Set Type: Prometheus

Set URL: http://thanos-querier:9090

Save and test. Now you can build your custom dashboard or simply import dashboard from grafana.net. Both Dashboard # 315 and # 1471 are ideal for getting started.

Deploy the Ingress object apiVersion: extensions/v1beta1kind: Ingressmetadata: name: monitoring-ingress namespace: monitoring annotations: kubernetes.io/ingress.class: "nginx" spec: rules:-host: grafana..com http: paths:-path: / backend: serviceName: grafana servicePort: 3000-host: prometheus-0..com http: paths:-path: / backend: ServiceName: prometheus-0-service servicePort: 8080-host: prometheus-1..com http: paths:-path: / backend: serviceName: prometheus-1-service servicePort: 8080-host: prometheus-2..com http: paths:-path: / backend: serviceName: prometheus-2-service servicePort: 8080-host: alertmanager..com http: paths:-path: / backend: serviceName: alertmanager servicePort: 9093-host: thanos-querier..com http: paths:-path: / backend: serviceName: thanos-querier servicePort: 9090-host: thanos-ruler..com http: paths:-path: / backend: serviceName: thanos-ruler servicePort: 9090

This is the last piece of the puzzle. It helps expose all our services outside the Kubernetes cluster and helps us access them. Be sure to replace it with a domain name that you can access, and you can point Ingress-Controller 's service to this domain name.

You should now be able to visit Thanos Querier at http://thanos-querier..com. It looks like this:

Make sure deduplication (deduplication) is selected.

If you click Store, you can see all the active endpoints discovered by the thanos-store-gateway service.

Now you can add Thanos Querier to Grafana as the data source and start creating dashboard.

Kubernetes cluster monitoring dashboard

Kubernetes node monitoring dashboard

Integrating Thanos with Prometheus undoubtedly provides the ability to scale out Prometheus, and because Thanos-Querier can extract metrics data from other querier instances, you can actually extract metrics data across clusters and visualize it in a single dashboard.

We can also archive metric data in object storage, providing unlimited storage space for our monitoring system, while providing metric data from the object storage itself. The main cost of this setup can be attributed to object storage (S3 or GCS). If we apply appropriate retention strategies to them, we can further reduce costs.

However, achieving all this requires a lot of configuration.

On how to use Prometheus and Thanos for high availability K8S monitoring is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.