Shulou(Shulou.com)06/02 Report--

Use the built-in cAdvisor monitoring container

CAdvisor is already built into the kubelet component, so we don't need to install it separately. The data path of cAdvisor is / api/v1/nodes//proxy/metrics.

1. Add job and update prometheus configuration

-job_name: 'kubernetes-cadvisor' kubernetes_sd_configs:-role: node scheme: https tls_config: ca_file: / var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: / var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs:-action: labelmap regex: _ _ meta_kubernetes_node_label_ (. +)-target_label: _ _ address__ replacement: kubernetes.default .svc: 443-source_labels: [_ _ meta_kubernetes_node_name] regex: (. +) target_label: _ _ metrics_path__ replacement: / api/v1/nodes/$ {1} / proxy/metrics/cadvisor

$kubectl apply-f prometheus-cm.yaml

$kubectl get svc-n kube-ops | grep prometheus

Prometheus NodePort 10.102.197.83 9090:32619/TCP

$curl-X POST "http://10.102.197.83:9090/-/reload" # to make the configuration effective

Monitoring apiserver

1. Add job and update prometheus configuration

-job_name: 'kubernetes-apiservers' kubernetes_sd_configs:-role: endpoints scheme: https tls_config: ca_file: / var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: / var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs:-source_labels: [_ _ meta_kubernetes_namespace, _ _ meta_kubernetes_service_name, _ _ meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https

$kubectl apply-f prometheus-cm.yaml

$kubectl get svc-n kube-ops | grep prometheus

Prometheus NodePort 10.102.197.83 9090:32619/TCP

$curl-X POST "http://10.102.197.83:9090/-/reload" # to make the configuration effective

Configure automatic discovery and monitoring of ordinary svc

1. Add job and update prometheus configuration

-job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs:-role: endpoints relabel_configs:-source_labels: [_ meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true-source_labels: [_ meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: _ scheme__ regex: (https?)-source_labels: [_ _ meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: _ _ metrics_path__ regex: (. +)-source_labels: [_ _ address__ _ _ meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: _ _ address__ regex: ([^:] +) (?::\ d +)? (\ d +) replacement: $1meta_kubernetes_service_label_ 2-action: labelmap regex: _ meta_kubernetes_service_label_ (. +)-source_labels: [_ meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace-source_labels: [_ meta_kubernetes_service_name] action: replace target_label: kubernetes_name

To automatically discover the Service in the cluster, we need to add a declaration of: prometheus.io/scrape=true to the annotation area of the Service

$kubectl apply-f prometheus-cm.yaml

$kubectl get svc-n kube-ops | grep prometheus

Prometheus NodePort 10.102.197.83 9090:32619/TCP

$curl-X POST "http://10.102.197.83:9090/-/reload" # to make the configuration effective

2. Modify the svc of redis to dynamically discover and monitor (on the basis of static discovery)

New annotations

Kind: ServiceapiVersion: v1metadata: name: redis namespace: kube-ops annotations: prometheus.io/scrape: "true" prometheus.io/port: "9121" spec: selector: app: redis ports:-name: redis port: 6379 targetPort: 6379-name: prom port: 9121 targetPort: 9121

Add the annotation of prometheus.io/scrape=true to the Service of redis created earlier.

Since the metrics interface of the redis service is on top of the 9121 redis-exporter service, we also need to add an annotations like prometheus.io/port=9121

$kubectl apply-f prome-redis.yaml

3. Modify the svc of trafik to dynamically discover and monitor (on the basis of static discovery)

ApiVersion: v1metadata: name: traefik-ingress-service namespace: kube-system annotations: prometheus.io/scrape: "true" # add prometheus.io/port: "8080" # add spec: selector: k8s-app: traefik-ingress-lb ports:-protocol: TCP port: 80 name: web-protocol: TCP port: 8080 name: admin type: NodePort

In the future, we have a new service, and if the service itself provides / metrics interface, we don't need to configure it statically at all.

When dynamic discovery of services is enabled, the following services are automatically discovered and monitored by default: prometheus itself, kube-dns

4. Automatically discover kube-state-metrics

Realize the state monitoring of various resource objects such as Pod, DaemonSet, Deployment, Job, CronJob and so on on the Kubernetes cluster

$git clone https://github.com/kubernetes/kube-state-metrics.git

$cd kube-state-metrics/kubernetes

$kubectl apply-f.

After deploying the kube-state-metrics to the Kubernetes, you will find that the Prometheus in the Kubernetes cluster will automatically discover the kube-state-metrics under the kubernetes-service-endpoints job and start pulling the metrics. This is because the manifest definition file kube-state-metrics-service.yaml that deploys the kube-state-metrics contains an annotation such as prometheus.io/scrape: 'true'.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.