Shulou(Shulou.com)06/01 Report--

This article is about how to use Prometheus-Operator to monitor Calico, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

The core component of Calico is Felix, which is responsible for setting routing tables, ACL rules, and so on, in order to provide the necessary network connection for the normal operation of endpoints resources on the host. It is also responsible for providing data about the health of the network (for example, reporting errors and problems in configuring its hosts), which is written to etcd to make it visible to other components and operators in the network.

Thus, for our monitoring, the core of monitoring Calico is that monitoring Felix,Felix is equivalent to the brain of Calico. In this article, you will learn how to use Prometheus-Operator to monitor Calico.

1. Configure Calico to enable metrics

The specification of Felix is disabled by default. You must manually change the Felix configuration through the command line management tool calicoctl to enable it. You need to configure the command line management tool in advance.

The Calico version used in this article is v3.15.0, and other versions are similar. Download the management tools first:

$wget https://github.com/projectcalico/calicoctl/releases/download/v3.15.0/calicoctl-O / usr/local/bin/calicoctl$ chmod + x / usr/local/bin/calicoctl

Next you need to set up the calicoctl configuration file (default is / etc/calico/calicoctl.cfg). If your Calico backend storage uses Kubernetes API, the configuration file is as follows:

ApiVersion: projectcalico.org/v3kind: CalicoAPIConfigmetadata:spec: datastoreType: "kubernetes" kubeconfig: "/ root/.kube/config"

If the Calico backend storage uses etcd, the configuration file is as follows:

ApiVersion: projectcalico.org/v3kind: CalicoAPIConfigmetadata:spec: datastoreType: "etcdv3" etcdEndpoints: https://192.168.57.51:2379,https://192.168.57.52:2379,https://192.168.57.53:2379 etcdKeyFile: / opt/kubernetes/ssl/server-key.pem etcdCertFile: / opt/kubernetes/ssl/server.pem etcdCACertFile: / opt/kubernetes/ssl/ca.pem

You need to change the certificate path to your etcd certificate path.

After you have configured calicoctl, you can view or modify the configuration of Calico. Let's take a look at the default Felix configuration:

$calicoctl get felixConfiguration default-o yamlapiVersion: projectcalico.org/v3kind: FelixConfigurationmetadata: creationTimestamp: "2020-06-25T14:37:28Z" name: default resourceVersion: "269031" uid: 52146c95-ff97-40a9-9ba7-7c3b4dd3ba57spec: bpfLogLevel: "" ipipEnabled: true logSeverityScreen: Info reportingInterval: 0s

You can see that metrics are not enabled in the default configuration. You need to modify the configuration manually. The command is as follows:

$calicoctl patch felixConfiguration default-patch'{"spec": {"prometheusMetricsEnabled": true}}'

The port of Felix exposure indicator is 9091. You can verify whether the metric is enabled by checking the listening port:

$ss-tulnp | grep 9091tcp LISTEN 0 4096 [:]: 9091 [::]: * users: ("calico-node", pid=13761 Fd=9) $curl-s http://localhost:9091/metrics# HELP felix_active_local_endpoints Number of active endpoints on this host.# TYPE felix_active_local_endpoints gaugefelix_active_local_endpoints "HELP felix_active_local_policies Number of active policies on this host.# TYPE felix_active_local_policies gaugefelix_active_local_policies" HELP felix_active_local_selectors Number of active selectors on this host.# TYPE felix_active_local_selectors gaugefelix_active_local_selectors 0..2. Felix index collected by Prometheus

Once the metrics of Felix are enabled, you can collect metrics data through Prometheus-Operator. During deployment, Prometheus-Operator creates five CRD resource objects, Prometheus, PodMonitor, ServiceMonitor, AlertManager and PrometheusRule, and then monitors and maintains the status of these five resource objects all the time. The resource object Prometheus is the abstraction of Prometheus Server. PodMonitor and ServiceMonitor are all kinds of abstractions of exporter, which are used to provide tools for providing index data interface. Prometheus uses the index data interface provided by PodMonitor and ServiceMonitor to pull data.

ServiceMonitor requires that the monitored service must have a corresponding Service, while PodMonitor does not. This paper chooses to use PodMonitor to collect Felix indicators.

Although PodMonitor does not need the application to create the corresponding Service, it must specify the port and name of the metric in the Pod, so you need to modify the configuration of DaemonSet calico-node first to specify the port and name. First open the configuration of DaemonSet calico-node with the following command:

$kubectl-n kube-system edit ds calico-node

Then modify it online and add the following to spec.template.sepc.containers:

Ports:-containerPort: 9091 name: http-metrics protocol: TCP

Create the PodMonitor corresponding to Pod:

# prometheus-podMonitorCalico.yamlapiVersion: monitoring.coreos.com/v1kind: PodMonitormetadata: labels: k8s-app: calico-node name: felix namespace: monitoringspec: podMetricsEndpoints:-interval: 15s path: / metrics port: http-metrics namespaceSelector: matchNames:-kube-system selector: matchLabels: k8s-app: calico-node$ kubectl apply-f prometheus-podMonitorCalico.yaml

There are several parameters to note:

The name of PodMonitor will eventually be reflected in the configuration of Prometheus as job_name.

The podMetricsEndpoints.port needs to be the same as the ports.name in the monitored Pod, in this case http-metrics.

The namespaceSelector.matchNames needs to be the same namespace as the monitored Pod, in this case kube-system.

The label of the selector.matchLabels must correspond to the tag that can be uniquely identified in the monitored Pod.

Finally, Prometheus-Operator will modify the configuration file of Prometheus according to PodMonitor to monitor the relevant Pod. You can open the UI of Prometheus to view the monitoring targets:

Notice that there is a pod= "calico-node-xxx" in Labels, indicating that you are monitoring Pod.

3. Visual monitoring index

After collecting the indicators, you can display the monitoring indicators through the dashboard of Grafana. The Grafana deployed in Prometheus-Operator cannot modify the configuration of the dashboard in real time (the json file of the dashboard must be mounted to Grafana Pod in advance), and it is not the latest version (version 7.0 or later), so I choose to delete the Grafana that comes with Prometheus-Operator and deploy the Grafana in the helm repository myself. First go to the manifests directory of the kube-prometheus project, then move all the deployment manifests related to Grafana to the same directory, and then delete Grafana:

$cd kube-prometheus/manifests$ mkdir grafana$ mv grafana-* grafana/$ kubectl delete-f grafana/

Then deploy the latest Grafana through helm:

$helm install grafana stable/grafana-n monitoring

The password for accessing Grafana is saved in Secret and can be viewed with the following command:

$kubectl-n monitoring get secret grafana-o yamlapiVersion: v1data: admin-password: MnpoV3VaMGd1b3R3TDY5d3JwOXlIak4yZ3B2cTU1RFNKcVY0RWZsUw== admin-user: YWRtaW4= ldap-toml: "" kind: Secretmetadata:...

Decrypt the password:

$echo-n "MnpoV3VaMGd1b3R3TDY5d3JwOXlIak4yZ3B2cTU1RFNKcVY0RWZsUw==" | base64-d

The decrypted information is the access password. The user name is admin. Log in to Grafana's UI with a username and password:

Add the data source for Prometheus-Operator:

Calico officially does not have a separate dashboard json, but puts it into the ConfigMap. We need to extract the json we need, extract the contents of the felix-dashboard.json, and then replace the datasource value with prometheus. You can replace it with sed or with editors. Most editors have the ability to replace globally. If you really don't know how to extract it, you can use the json I extracted:

After the modification, import the json content into Grafana:

The resulting Felix dashboard is shown in the following figure:

Kubernetes 1.18.2 1.17.5 1.16.9 1.15.12 offline installation package release address http://store.lameleg.com, welcome to experience. The latest version of sealos v3.3.6 is used. The host name resolution configuration is optimized, lvscare mount / lib/module solves the boot ipvs loading problem, fixes the incompatibility between lvscare community netlink and 3.10 kernel, sealos generates century-old certificate and so on. More features https://github.com/fanux/sealos. Welcome to scan the QR code below to join the nail group, the nail group has been integrated with sealos robots can see the dynamics of sealos in real time.

The above is how to use Prometheus-Operator to monitor Calico. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.