Shulou(Shulou.com)06/02 Report--

Monitoring sharp weapon prometheus helm deployment configuration test

1. Deploy helm

Deploy the helm reference method

Later, helm is used to deploy grafana and prometheus, so you need to deploy helm first to ensure that helm can be used properly.

The process of deploying a helm client is as follows:

[root@k8s-node1 helm] # curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh% Total% Received% Xferd Average Speed Time Current Dload Upload Total Spent Left Speed100 6617 6617 00 5189 00: 00:01 0:00:01 -:-5193 [root@k8s -node1 helm] # lsget_ helm.sh [root @ k8s-node1 helm] # chmod 700 get_ helm.sh [root @ k8s-node1 helm] #. / get_helm.sh Downloading https://get.helm.sh/helm-v3.0.2-linux-amd64.tar.gzPreparing to install helm into / usr/local/binhelm installed into / usr/local/bin/helm [root@k8s-node1 helm] # helm versionversion.BuildInfo {Version: "v3.0.2" GitCommit: "19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState: "clean", GoVersion: "go1.13.5"}

Add yum

[root@k8s-node1 helm] # helm repo add stable https://kubernetes-charts.storage.googleapis.com/"stable" has been added to your repositories

Search for it.

[root@k8s-node1 helm] # helm search repo stable | grep grafanastable/grafana 4.2.2 6.5.2 The leading tool for querying and visualizing t... [root@k8s-node1 helm] # helm search repo stable | grep prometheusstable/helm-exporter 0.3.1 0.4.0 Exports helm release stats To prometheus stable/prometheus 9.7.2 2.13.1 Prometheus is a monitoring system and time seri...stable/prometheus-adapter 1.4.0 v0.5.0 A Helm chart for k8s prometheus adapter stable/prometheus-blackbox-exporter 1.6.0 0.15.1 Prometheus Blackbox Exporter stable/prometheus-cloudwatch-exporter 0.5.0 0.6.0 A Helm chart for prometheus cloudwatch-exporter stable/prometheus-consul-exporter 0.1.4 0.4.0 A Helm chart for the Prometheus Consul Exporter stable/prometheus-couchdb-exporter 0 . 1.1 1.0 A Helm chart to export the metrics from couchdb...stable/prometheus-mongodb-exporter 2.4.0 v0.10.0 A Prometheus exporter for MongoDB metrics stable/prometheus-mysql-exporter 0.5.2 v0.11.0 A Helm chart for prometheus mysql exporter with... Stable/prometheus-nats-exporter 2.3.0 0.6.0 A Helm chart for prometheus-nats-exporter stable/prometheus-node-exporter 1.8.1 0.18.1 A Helm chart for prometheus node-exporter stable/prometheus-operator 8.5.0 0.34.0 Provides easy monitoring definitions for Kubern...stable/prometheus-postgres-exporter 1.1.1 0.5.1 A Helm chart for prometheus postgres-exporter stable/prometheus-pushgateway 1.2.10 1.0.1 A Helm chart for prometheus pushgateway stable/prometheus-rabbitmq-exporter 0.5.5 V0.29.0 Rabbitmq metrics exporter for prometheus stable/prometheus-redis-exporter 3.2.0 1.0.4 Prometheus exporter for Redis metrics stable/prometheus-snmp-exporter 0.0.4 0.14.0 Prometheus SNMP Exporter stable / prometheus-to-sd 0.3.0 0.5.2 Scrape metrics stored in prometheus format and...

Deploy an application test

[root@k8s-node1 helm] # helm install stable/nginx-ingress-- generate-nameNAME: nginx-ingress-1577092943LAST DEPLOYED: Mon Dec 23 17:22:26 2019NAMESPACE: defaultSTATUS: deployedREVISION: 1TEST SUITE: NoneNOTES:The nginx-ingress controller has been installed.It may take a few minutes for the LoadBalancer IP to be available.You can watch the status by running 'kubectl-- namespace default get services-o wide-w nginx-ingress-1577092943-controller' [root@k8s-node1 helm] # helm lsNAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSIONnginx-ingress-1577092943 default 1 2019-12-23 17 22 26.230661264 + 0800 0.26.1

Everyone's up. Let's meet.

[root@k8s-node1 helm] # kubectl get all | grep nginxpod/nginx-ingress-1577092943-controller-8468884448-9wszl 1 Running 0 4m49spod/nginx-ingress-1577092943-default-backend-74c4db5b5b-clc2s 1 Running 0 4m49sservice/nginx-ingress-1577092943-controller LoadBalancer 10.254.229.168 80:8691/TCP 443:8569/TCP 4m49sservice/nginx-ingress-1577092943-default-backend ClusterIP 10.254.37.89 80/TCP 4m49sdeployment.apps/nginx-ingress-1577092943-controller 1/1 1 1 4m49sdeployment.apps/nginx-ingress-1577092943-default-backend 1/1 1 1 4m49sreplicaset.apps/nginx-ingress-1577092943-controller -8468884448 1 1 1 4m49sreplicaset.apps/nginx-ingress-1577092943-default-backend-74c4db5b5b 1 1 1 4m49s

Deployment is complete, testing is feasible, and remove currently installed applications.

[root@k8s-node1 helm] # helm lsNAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSIONnginx-ingress-1577092943 default 1 2019-12-23 17 CST deployed nginx-ingress-1.27.0 22 CST deployed nginx-ingress-1.27.0 26.230661264 + 0800 helm uninstall nginx-ingress-1577092943release "nginx-ingress-1577092943" uninstalled

2.helm deployment prometheus

Helm deployment prometheus

Official address of Prometheus

Prometheus learning documentation

2.1. Start deployment

[root@k8s-node1] # helm install stable/prometheus-- generate-nameNAME: prometheus-1577239571LAST DEPLOYED: Wed Dec 25 10:06:14 2019NAMESPACE: defaultSTATUS: deployedREVISION: 1TEST SUITE: NoneNOTES:The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:prometheus-1577239571-server.default.svc.cluster.local

2.2. Encounter problems

Retrieve the started svc,pod

[root@k8s-node1 ~] # kubectl get svc,pod-o wide | grep prometheusservice/prometheus-1577239571-alertmanager ClusterIP 10.254.251.30 80/TCP 2m26s app=prometheus,component=alertmanager,release=prometheus-1577239571service/prometheus-1577239571-kube-state-metrics ClusterIP None 80/TCP 2m26s app=prometheus,component=kube-state-metrics Release=prometheus-1577239571service/prometheus-1577239571-node-exporter ClusterIP None 9100/TCP 2m26s app=prometheus,component=node-exporter,release=prometheus-1577239571service/prometheus-1577239571-pushgateway ClusterIP 10.254.188.166 9091/TCP 2m26s app=prometheus,component=pushgateway,release=prometheus-1577239571service/prometheus-1577239571-server ClusterIP 10.254.128.74 80/TCP 2m26s app=prometheus,component=server Release=prometheus-1577239571pod/prometheus-1577239571-alertmanager-67b967b8c7-lmjf7 0/2 Pending 0 2m25s pod/prometheus-1577239571-kube-state-metrics-6d86bf588b-w7hrq 1/1 Running 0 2m25s 172.30.4.7 k8s-node1 pod/prometheus-1577239571-node-exporter-k9bsf 1/1 Running 0 2m25s 192.168.174.130 k8s-node3 pod/prometheus-1577239571-node-exporter-rv9k8 1 k8s-node3 pod/prometheus-1577239571-node-exporter-rv9k8 1 Running 0 2m25s 192.168.174.129 k8s-node2 pod/prometheus-1577239571-node-exporter-xc8f2 1 Running 0 2m25s 192.168.174.128 k8s- Node1 pod/prometheus-1577239571-pushgateway-d9b4cb944-zppfm 1/1 Running 0 2m25s 172.30.26.7 k8s-node3 pod/prometheus-1577239571-server-c5d4dffbf-gzk9n 0/2 Pending 0 2m25s

There are two pod that have always been pending status. Search the reason. Check the describe to see the following error:

Warning FailedScheduling 25s (x5 over 4m27s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 3 times)

Error report of pvc, retrieve pvc

[root@k8s-node1 templates] # kubectl get pvc | grep prometheusprometheus-1577239571-alertmanager Pending 21mprometheus-1577239571-server Pending 21m

Describe pvc takes a look at the details and reports an error. There is no pv or no interfacing storage, so pvc cannot be enabled. Please see below:

Normal FailedBinding 16s (x82 over 20m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set

What shall I do? My cluster here is stored by pvc dynamic docking nfs. Can it be modified to dock with nfs storage?

Refer to the previous article for interfacing with Nfs storage. The name of storageclass is as follows:

[root@k8s-node1 templates] # kubectl get storageclassNAME PROVISIONER AGEmanaged-nfs-storage fuseim.pri/ifs 5d17h

2.3. Docking storage to solve error reporting

Retrieve the variable of stable/prometheus and the variable setting of the pv that reported an error. Refer to the command below.

Helm show values stable/prometheus

Because you need to modify after retrieval, download the chart file and retrieve and modify it.

[root@k8s-node1 prometheus-grafana] # helm pull stable/prometheus [root@k8s-node1 prometheus-grafana] # lsprometheus-9.7.2.tgz [root@k8s-node1 prometheus-grafana] # tar zxvf prometheus-9.7.2.tgz-- warning=no-timestamp [root@k8s-node1 prometheus-grafana] # lsprometheus prometheus-9.7.2.tgz [root@k8s-node1 prometheus-grafana] # tree prometheusprometheus ├── Chart.yaml ├── README.md ├── templates │ ├── Alertmanager-clusterrolebinding.yaml │ ├── alertmanager-clusterrole.yaml │ ├── alertmanager-configmap.yaml │ ├── alertmanager-deployment.yaml │ ├── alertmanager-ingress.yaml │ ├── alertmanager-networkpolicy.yaml │ ├── alertmanager-pdb.yaml │ ├── alertmanager-podsecuritypolicy.yaml │ ├── alertmanager-pvc.yaml │ ├── alertmanager-serviceaccount.yaml │ alertmanager-service-headless.yaml ├── alertmanager-service.yaml │ ├── alertmanager-statefulset.yaml │ ├── _ helpers.tpl │ ├── kube-state-metrics-clusterrolebinding.yaml │ ├── kube-state-metrics-clusterrole.yaml │ ├── kube-state-metrics-deployment.yaml │ ├── kube-state-metrics-networkpolicy.yaml │ ├── kube-state-metrics-pdb.yaml │ ├── kube-state-metrics-podsecuritypolicy.yaml │ ├── kube-state-metrics-serviceaccount.yaml │ ├── kube-state-metrics-svc.yaml │ ├── node-exporter-daemonset.yaml │ ├── node-exporter-podsecuritypolicy.yaml │ ├── node-exporter-rolebinding.yaml │ ├── node-exporter-role.yaml │ ├── node-exporter-serviceaccount.yaml │ ├── node-exporter-service.yaml │ ├── NOTES.txt │ ├── pushgateway-clusterrolebinding.yaml │ ├── pushgateway-clusterrole.yaml │ ├── pushgateway-deployment.yaml │ ├── pushgateway-ingress.yaml │ ├── pushgateway-networkpolicy.yaml │ ├── pushgateway-pdb.yaml │ ├── pushgateway-podsecuritypolicy.yaml │ ├── pushgateway-pvc.yaml │ ├── pushgateway-serviceaccount.yaml │ ├── pushgateway-service.yaml server-clusterrolebinding.yaml ├── server-clusterrole.yaml │ ├── server-configmap.yaml │ ├── server-deployment.yaml │ ├── server-ingress.yaml │ ├── server-networkpolicy.yaml │ ├── server-pdb.yaml │ ├── server-podsecuritypolicy.yaml │ ├── server-pvc.yaml │ ├── server-serviceaccount.yaml │ ├── server-service-headless.yaml server-service. Yaml │ ├── server-statefulset.yaml │ └── server-vpa.yaml └── values.yaml1 directory 56 files

The file that defines all variables is the values.yaml file, which is retrieved.

It contains a lot of things and needs to be checked one by one, one of the configuration definitions of pv.

PersistentVolume: # # If true, alertmanager will create/use a Persistent Volume Claim # # If false Use emptyDir # # enabled: true # # alertmanager data Persistent Volume access modes # # Must match those of existing PV or dynamic provisioner # # Ref: http://kubernetes.io/docs/user-guide/persistent-volumes/ # # accessModes:-ReadWriteOnce # # alertmanager data Persistent Volume Claim annotations # # annotations: {} # # alertmanager data Persistent Volume existing claim name # # Requires alertmanager.persistentVolume.enabled: true # # If defined PVC must be created manually before volume will be bound existingClaim: "# # alertmanager data Persistent Volume mount root path # # mountPath: / data # # alertmanager data Persistent Volume size # # size: 2Gi # # alertmanager data Persistent Volume Storage Class # # If defined, storageClassName: # # If set to"-", storageClassName:", which disables dynamic provisioning # # If undefined (the default) or set to null, no storageClassName spec is # # set, choosing the default provisioner. (gp2 on AWS, standard on # # GKE, AWS & OpenStack) # storageClass: "-" # # alertmanager data Persistent Volume BindingMode # # If defined, volumeBindingMode: # # If undefined (the default) or set to null, no volumeBindingMode spec is # # set, choosing the default mode. # #

According to the above variable explanation, chart defines a pvc of 2GB. The parameter to configure pvc docking dynamic storage is: storageClass, which is not enabled by default. Enable this parameter to dock with storageclass.

Change # storageClass: "-" to storageClass: managed-nfs-storage (managed-nfs-storage is the name of the storageclass I configured in the cluster, which needs to be modified in total)

[root@k8s-node1 prometheus-grafana] # cat prometheus/values.yaml | grep-B 8 managed # # alertmanager data Persistent Volume Storage Class # # If defined, storageClassName: # # If set to "-", storageClassName: ", which disables dynamic provisioning # # If undefined (the default) or set to null, no storageClassName spec is # # set, choosing the default provisioner. (gp2 on AWS, standard on # # GKE, AWS & OpenStack) # storageClass: "-" storageClass: managed-nfs-storage-- # # Prometheus server data Persistent Volume Storage Class # # If defined, storageClassName: # # If set to "-", storageClassName: ", which disables dynamic provisioning # # If undefined (the default) or set to null, no storageClassName spec is # # set, choosing the default provisioner. (gp2 on AWS, standard on # # GKE, AWS & OpenStack) # storageClass: "-" storageClass: managed-nfs-storage-# # pushgateway data Persistent Volume Storage Class # # If defined, storageClassName: # # If set to "-", storageClassName: ", which disables dynamic provisioning # # If undefined (the default) or set to null, no storageClassName spec is # # set, choosing the default provisioner. (gp2 on AWS, standard on # # GKE, AWS & OpenStack) # storageClass: "-" storageClass: managed-nfs-storage

Yes, after modifying the docking storage parameters, the installation is successful, see below

[root@k8s-node1 prometheus-grafana] # kubectl get svc,pod-o wide | grep prometheusservice/prometheus-1577263826-alertmanager ClusterIP 10.254.112.105 80/TCP 4m6s app=prometheus,component=alertmanager,release=prometheus-1577263826service/prometheus-1577263826-kube-state-metrics ClusterIP None 80/TCP 4m6s app=prometheus,component=kube-state-metrics Release=prometheus-1577263826service/prometheus-1577263826-node-exporter ClusterIP None 9100/TCP 4m6s app=prometheus,component=node-exporter,release=prometheus-1577263826service/prometheus-1577263826-pushgateway ClusterIP 10.254.185.145 9091/TCP 4m6s app=prometheus,component=pushgateway,release=prometheus-1577263826service/prometheus-1577263826-server ClusterIP 10.254.132.104 80/TCP 4m6s app=prometheus,component=server Release=prometheus-1577263826pod/prometheus-1577263826-alertmanager-5cfccc55b7-6hdqn 2 Running 2 Running 0 4m5s 172.30.26.8 k8s-node3 pod/prometheus-1577263826-kube-state-metrics-697db589d4-d5rmm 1 Running 0 4m5s 172.30.26.7 k8s-node3 pod/prometheus-1577263826-node-exporter-5gcc2 1 / 1 Running 0 4m5s 192.168.174.129 k8s-node2 pod/prometheus-1577263826-node-exporter-b569p 1/1 Running 0 4m5s 192.168.174.130 k8s-node3 pod/prometheus-1577263826-node-exporter-mft6l 1/1 Running 0 4m5s 192.168.174 . 128 k8s-node1 pod/prometheus-1577263826-pushgateway-95c67bd5d-28p25 1/1 Running 0 4m5s 172.30.4.7 k8s-node1 pod/prometheus-1577263826-server-88fbdfc47-p2bfm 2/2 Running 0 4m5s 172.30.4.8 k8s-node1

Basic concepts of 2.4.prometheus

The role of these components of prometheus

Data source

Prometheus server

Prometheus Server is the core part of Prometheus component, which is responsible for obtaining, storing and querying monitoring data.

Prometheus Server built-in Express Browser UI, through this UI can directly through the PromQL data query and visualization.

Node-exporter

Exporter exposes the endpoint of monitoring data collection to Prometheus Server,Prometheus Server in the form of HTTP service. By accessing the Endpoint endpoint provided by the Exporter, the monitoring data to be collected can be obtained.

Alertmanager

The creation of alarm rules based on PromQL is supported in Prometheus Server. If the rules defined by PromQL are met, an alarm will be generated, and the subsequent processing flow of the alarm will be managed by AlertManager. In AlertManager, we can integrate with email, Slack and other built-in notification methods, or customize alarm handling methods through Webhook. AlertManager is the alarm processing center in the Prometheus system.

Pushgateway

Because the design of Prometheus data acquisition is based on Pull model, it is necessary for Prometheus Server to communicate with Exporter directly in the configuration of network environment. When this kind of network demand can not be met directly, PushGateway can be used for transit. The monitoring data of the internal network can be actively Push to Gateway through PushGateway. On the other hand, Prometheus Server can obtain monitoring data from PushGateway in the same Pull way.

This is not needed in the environment here.

Kube-state-metrics

The basic concept is that kube-state-metrics polls Kubernetes API and converts the structured information of Kubernetes into metrics. For example, how many rc are scheduled and how many rc are available now? How many Job are being executed now?

2.5. Configure web to access prometheus server and kube-state-metrics

Traefik has been deployed in the previous environment. You only need to add ingress, as shown below:

Prometheus server

[root@k8s-node1 prometheus-grafana] # cat prometheus-server-ingress.yaml apiVersion: extensions/v1beta1kind: Ingressmetadata: name: prometheus-server namespace: defaultspec: rules:-host: prometheus-server http: paths:-path: / backend: serviceName: prometheus-1577263826-server servicePort: 80

Kube-state-metrics

[root@k8s-node1 prometheus-grafana] # cat kube-state-ingress.yaml apiVersion: extensions/v1beta1kind: Ingressmetadata: name: kube-state namespace: defaultspec: rules:-host: kube-state http: paths:-path: / backend: serviceName: prometheus-1577263826-kube-state-metrics servicePort: 80

Specify Host resolution for normal access. Note that both server are https. How to configure traefik, please refer to the configuration of traefik.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.