Monitor containers, apiserver, and auto-discover and monitor services with prometheus
Use the built-in cAdvisor monitoring container
CAdvisor is already built into the kubelet component, so we don't need to install it separately. The data path of cAdvisor is / api/v1/nodes//proxy/metrics.
1. Add job and update prometheus configuration
-job_name: 'kubernetes-cadvisor' kubernetes_sd_configs:-role: node scheme: https tls_config: ca_file: / var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: / var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs:-action: labelmap regex: _ _ meta_kubernetes_node_label_ (. +)-target_label: _ _ address__ replacement: kubernetes.default .svc: 443-source_labels: [_ _ meta_kubernetes_node_name] regex: (. +) target_label: _ _ metrics_path__ replacement: / api/v1/nodes/$ {1} / proxy/metrics/cadvisor
$kubectl apply-f prometheus-cm.yaml
$kubectl get svc-n kube-ops | grep prometheus
Prometheus NodePort 10.102.197.83 9090:32619/TCP
$curl-X POST "http://10.102.197.83:9090/-/reload" # to make the configuration effective
Monitoring apiserver
1. Add job and update prometheus configuration
-job_name: 'kubernetes-apiservers' kubernetes_sd_configs:-role: endpoints scheme: https tls_config: ca_file: / var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: / var/run/secrets/kubernetes.io/serviceaccount/token relabel_configs:-source_labels: [_ _ meta_kubernetes_namespace, _ _ meta_kubernetes_service_name, _ _ meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https
$kubectl apply-f prometheus-cm.yaml
$kubectl get svc-n kube-ops | grep prometheus
Prometheus NodePort 10.102.197.83 9090:32619/TCP
$curl-X POST "http://10.102.197.83:9090/-/reload" # to make the configuration effective
Configure automatic discovery and monitoring of ordinary svc
1. Add job and update prometheus configuration
-job_name: 'kubernetes-service-endpoints' kubernetes_sd_configs:-role: endpoints relabel_configs:-source_labels: [_ meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true-source_labels: [_ meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: _ scheme__ regex: (https?)-source_labels: [_ _ meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: _ _ metrics_path__ regex: (. +)-source_labels: [_ _ address__ _ _ meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: _ _ address__ regex: ([^:] +) (?::\ d +)? (\ d +) replacement: $1meta_kubernetes_service_label_ 2-action: labelmap regex: _ meta_kubernetes_service_label_ (. +)-source_labels: [_ meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace-source_labels: [_ meta_kubernetes_service_name] action: replace target_label: kubernetes_name
To automatically discover the Service in the cluster, we need to add a declaration of: prometheus.io/scrape=true to the annotation area of the Service
$kubectl apply-f prometheus-cm.yaml
$kubectl get svc-n kube-ops | grep prometheus
Prometheus NodePort 10.102.197.83 9090:32619/TCP
$curl-X POST "http://10.102.197.83:9090/-/reload" # to make the configuration effective
2. Modify the svc of redis to dynamically discover and monitor (on the basis of static discovery)
New annotations
Kind: ServiceapiVersion: v1metadata: name: redis namespace: kube-ops annotations: prometheus.io/scrape: "true" prometheus.io/port: "9121" spec: selector: app: redis ports:-name: redis port: 6379 targetPort: 6379-name: prom port: 9121 targetPort: 9121
Add the annotation of prometheus.io/scrape=true to the Service of redis created earlier.
Since the metrics interface of the redis service is on top of the 9121 redis-exporter service, we also need to add an annotations like prometheus.io/port=9121
$kubectl apply-f prome-redis.yaml
3. Modify the svc of trafik to dynamically discover and monitor (on the basis of static discovery)
ApiVersion: v1metadata: name: traefik-ingress-service namespace: kube-system annotations: prometheus.io/scrape: "true" # add prometheus.io/port: "8080" # add spec: selector: k8s-app: traefik-ingress-lb ports:-protocol: TCP port: 80 name: web-protocol: TCP port: 8080 name: admin type: NodePort
In the future, we have a new service, and if the service itself provides / metrics interface, we don't need to configure it statically at all.
When dynamic discovery of services is enabled, the following services are automatically discovered and monitored by default: prometheus itself, kube-dns
4. Automatically discover kube-state-metrics
Realize the state monitoring of various resource objects such as Pod, DaemonSet, Deployment, Job, CronJob and so on on the Kubernetes cluster
$git clone https://github.com/kubernetes/kube-state-metrics.git
$cd kube-state-metrics/kubernetes
$kubectl apply-f.
After deploying the kube-state-metrics to the Kubernetes, you will find that the Prometheus in the Kubernetes cluster will automatically discover the kube-state-metrics under the kubernetes-service-endpoints job and start pulling the metrics. This is because the manifest definition file kube-state-metrics-service.yaml that deploys the kube-state-metrics contains an annotation such as prometheus.io/scrape: 'true'.
