Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to monitor the performance of Kubernetes through Splunk. It is very detailed and has a certain reference value. Friends who are interested must finish it!

Splunk is the industry's leading platform for machine data collection, processing, analysis and management. Machine data refers to the data produced by applications, servers, storage, network equipment, security equipment and other related equipment that can provide power for the development of enterprise business. Machine data can reflect the behavior or activities of customers, transactions, applications, servers, networks, etc. The scope of machine data goes far beyond logs.

Deploy Architectur

The following figure shows the deployment architecture of the solution, which mainly includes:

Use Heapster to collect performance data of K8s, including CPU,Memory,Network,File System, etc.

Use the Statsd Sink of Heapster to send data to the Metrics Store of Splunk

Use Splunk search commands and dashboard functions to monitor performance data

Preparation in advance

There are two main things to be prepared in the early stage:

Compile the latest Heapster image and upload it to a public Docker image repository, such as docker hub

Configure Metrics Store and corresponding network input (Network Input UDP/TCP) in Splunk

The main choice here is whether Statsd's transport protocol should be UDP or TCP. I recommend using TCP here. The latest Heapster code supports different Backend, including log, influxdb, stackdriver, gcp monitoring, gcp logging, statsd, hawkular-metrics, wavefront, openTSDB, kafka, riemann, elasticsearch, and so on. Because Splunk's Metrics Store supports the statsd protocol, it can be easily integrated with Heapster.

First, we need to compile a container image with the latest heapster code, because the official version of the heapsterd image on docker hub is older and does not support statsd. So you need to compile it yourself.

Mkdir myheapstermkdir myheapster/srcexport GOPATH=myheapstercd myheapster/srcgit clone https://github.com/kubernetes/heapster.gitcd heapstermake container

Run the above command to compile the latest heapster image.

Note that heapster uses the udp protocol by default. If you want to use tcp, you need to modify the code.

Https://github.com/kubernetes/heapster/blob/master/metrics/sinks/statsd/statsd_client.go

Func (client * statsdClientImpl) open () error {var err errorclient.conn, err = net.Dial ("udp", client.host) if err! = nil {glog.Errorf ("Failed to open statsd client connection:% v", err)} else {glog.V (2). Infof ("statsd client connection opened:% + v", client.conn)} return err}

Change udp to tcp.

I put two images on docker hub, corresponding to the udp version of tcp, which you can use directly.

Naughtytao/heapster-amd64:v1.5.0-beta.3 udp

Naughtytao/heapster-amd64:v1.5.0-beta.4 tcp

Then you need to configure Metrics Store in Splunk, refer to this document

Install and configure Heapster

It's easy to deploy heapster on K8s, just create the corresponding yaml configuration file, and then create it with the kubectl command line.

The following are the configuration files for Deployment and Service:

Deployment.yaml

ApiVersion: extensions/v1beta1kind: Deploymentmetadata: name: heapster namespace: kube-systemspec: replicas: 1 template: metadata: labels: task: monitoring k8s-app: heapster version: V6 spec: containers:-name: heapster image: naughtytao/heapster-amd64:v1.5.0-beta.3 imagePullPolicy: Always command:-/ heapster- source=kubernetes: https://kubernetes.default -sink=statsd:udp://ip:port?numMetricsPerMsg=1

Service.yaml

ApiVersion: v1kind: Servicemetadata: labels: task: monitoring # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line. Kubernetes.io/cluster-service: 'true' kubernetes.io/name: Heapster name: heapster namespace: kube-systemspec: ports:-port: 80 targetPort: 8082 selector: k8s-app: heapster

Note the configuration of-sink for deployment. Ip is the IP or hostname of Splunk, and the port number of data input of Splunk corresponding to port. When using the udp protocol, the value of numMetricsPerMsg that needs to be configured is relatively small. When this value is relatively large, the error of message too long will be given. You can configure larger values when using tcp.

Run kubectl apply-f * .yaml to deploy heapster

If it is running normally, the corresponding heapster pod log is as follows

I0117 18 https://kubernetes.default 10 Using Kubernetes client with master 56.054746 1 heapster.go:78] / heapster-- source=kubernetes: https://kubernetes.default-- sink=statsd:udp://ec2-34-203-25-154.compute-1.amazonaws.com:8124?numMetricsPerMsg=10I0117 18 configs.go:61 10 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" And version v1I0117 18 and version v1I0117 10 driver.go:104 56.054978 1 driver.go:104] statsd metrics sink using configuration: {host:ec2-34-203-25-154.compute-1.amazonaws.com:8124 prefix: numMetricsPerMsg:10 protocolType:etsystatsd renameLabels:map [] allowedLabels:map [] customizeLabel:0x15fc8c0} I0117 18 driver.go:104 10 driver.go:104] statsd metrics sink using configuration: {host: Ec2-34-203-25-154.compute-1.amazonaws.com:8124 prefix: numMetricsPerMsg:10 protocolType:etsystatsd renameLabels:map [] allowedLabels:map [] customizeLabel:0x15fc8c0} I0117 18 customizeLabel:0x15fc8c0 10 allowedLabels:map 56.076272 1 heapster.go:202] Starting with StatsD SinkI0117 1815 10 numMetricsPerMsg:10 protocolType:etsystatsd renameLabels:map 56.076281 heapster.go:202] Starting with Metric SinkI0117 18 10 heapster.go:112 56.090229 1 heapster.go:112] Starting heapster on port 8082 monitors in Splunk

Well, if everything goes well, heapster will send metrics to Splunk's metrics store using the protocol and format of statsd.

The mstats and mcatalog commands of SPL can then be used to analyze and monitor metrics data.

The following search statement lists all Metrics

| | mcatalog values (metric_name) |

The following search statement lists the CPU usage of the entire cluster, and we can use Area or Line Chart to visualize the search results.

| | mstats avg (_ value) WHERE metric_name=cluster.cpu/usage_rate span=30m |

Corresponding memory usage of kube-system namespace

| | mstats avg (_ value) WHERE metric_name=namespace.kube-system.memory/usage span=30m |

You can put the analysis results that you are interested in in Dashboard and use the Realtime settings to monitor.

These are all the contents of the article "how to monitor the performance of Kubernetes through Splunk". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.