Shulou(Shulou.com)06/01 Report--

Based on how to realize the application-level monitoring of micro-services in kubernetes, it is believed that many inexperienced people do not know what to do. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

The Challenge of Micro Service Monitoring

The purpose of monitoring is to make all the service components in the cluster, whether they are HTTP services, database services, or middleware services. Can run healthily and stably, can find the problem, encounter the problem can find the reason.

In the past, monitoring tools focused on infrastructure or a single software component and on measuring operational health. These tools have achieved only some success in achieving this goal, but they work well for single, traditional applications and infrastructure. The emergence of microservices exposes weaknesses in tools.

Components are now hosted in a virtualized machine or container located between a private cloud, a public cloud, or a mixture of the two. I learned that I don't need to care about how much the service cpu uses and how much memory is used. Ensuring that these services communicate with each other to provide the desired results requires several important things from a monitoring perspective:

Is the throughput and response time of all services in the microservice cluster normal?

Which lines in the service invocation line are overloaded and which are underloaded?

The error rate of the service, such as HTTP 500error.

We want to monitor the analytics application, is it more direct to start from the status of its service?

There is a monitoring scheme.

At present, some manufacturers have proposed monitoring solutions for micro-services.

Monitor the end-to-end status of the service from an APM perspective.

Agent is developed for each type of service to collect application status information.

By generating a unified application log analysis monitoring scheme

Other options

Every commercial or open source solution has its advantages. You can choose according to your needs. For example, all your services are developed by yourself, and the log standards are consistent and can be handled uniformly by or. All access information can be logged, so I think log analysis is probably the most suitable solution for you. But for public cloud platforms, that's different.

The scheme adopted by the good Rain Cloud gang

The good Rain Cloud gang provides public cloud and privatized deployment, with a variety of services deployed within the platform. Various communication protocols and log standards. How do we monitor the application status of all services? Good Rain Cloud help improve the tenant network, environment isolation, so we provide users in their own environment to install their own monitoring components, our basic data collection is through network analysis. The following is explained in detail:

Kubernetes POD sharing mechanism

Containers in pod in kubernetes share cyberspace and hang on volumes to provide convenience for us to monitor the main service containers in pod. In fact, in terms of the use of the official definition of pod:

* content management systems, file and data loaders, local cache managers, etc.* log and checkpoint backup, compression, rotation, snapshotting, etc.* data change watchers, log tailers, logging and monitoring adapters, event publishers, etc.* proxies, bridges, and adapters* controllers, managers, configurators, and updaters

We can deploy some ancillary services in pod in addition to the main service. In the previous article, I talked about using pod's plug-in service to collect and process logs. Today I'm going to talk about using pod's network convenience to monitor the application-level metrics of the main service.

Analyze the application status of network traffic response by grabbing packets with shared network cards

Let's take a http service as an example. We monitor network traffic and get almost all access and service return information. For example, how many request in a minute, which path are requested, and how long the service has returned. Return the status code and other information.

To obtain the above data, we need to obtain the network packet, decode the network packet and then obtain the http protocol data.

Our WatchData service container and the application container are in the same pod. Through the traffic of the eth0 Nic in the application container, we obtain the traffic from the eth0 Nic in the WatchData container. The key information of the http header is obtained by decoding the network packet, and each Response forms a message sent to the server to complete the analysis, and then stores and forms continuous real-time monitoring data. The following figure shows a brief picture of the overall architecture:

Of course, as mentioned above, the main purpose of this solution is to be able to monitor various applications, but how can http work. Impassable communication applications use different communication protocols, such as mysql protocol and mongodb protocol. The TCP/IP network protocol stack is divided into four layers: application layer (Application), transport layer (Transport), network layer (Network) and link layer (Link). The network packet information we captured is also a four-layer model.

Implementation of Network packet capture Golang

It is very easy to use golang to capture packets on the network. Thanks to Google's package:

Github.com/google/gopacketgithub.com/google/gopacket/layersgithub.com/google/gopacket/pcap

Here I will give a Demo main code for listening to the network card.

/ / device Nic name if handle, err: = pcap.OpenLive (device, int32 (n.Option.Snaplen), true, n.Option.TimeOut); err! = nil {log.With ("error", err.Error ()) .Errorln ("PCAP OpenLive Error.") Return 1} else if err: = handle.SetBPFFilter (n.Option.Expr) Err! = nil {/ / optional log.With ("error", err.Error ()) .Errorln ("PCAP SetBPFFilter Error.", n.Option.Expr) return 1} else {log.Infoln ("Start listen the device", device) packetSource: = gopacket.NewPacketSource (handle) Handle.LinkType () go func (close chan struct {}, h * pcap.Handle) {for {select {case packet: =

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.