Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the principle of Docker data collection". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let Xiaobian take you to learn "What is the principle of Docker data collection"!

monitoring purposes

reduce downtime

Scale and performance management

resource plan

Identify abnormal events

Troubleshooting, analysis

Why monitor our services? There are obvious reasons for this, such as the need for monitoring tools to alert service failures, such as by monitoring service load to decide whether to scale up or down. If the general load of the machine is not high, you can consider reducing the size of the machine. If the database connection is often maintained at a high level, you can consider whether it can be dismantled and optimize the architecture.

##Docker Monitoring Challenges

Docker Features

Like a host, but not a host.

technology, China jiliang

Short life cycle monitoring blind spot (fault)

Microservices cluster

all-round

Host（VM） + Services + Containers + Apps

Containers bring more direction and possibilities to our development and operations, and we need a modern monitoring solution to cope with this change.

With the popularity of the immutable infrastructure concept and the rise of cloud-native applications, cloud computing components have become more and more like building blocks for toys. A lot of infrastructure life cycles become shorter, not only containers, but also cloud hosts and VMs.

Before cloud computing, a machine might last three, five or more years without needing to be reinstalled and the hostname would not change, but now it is possible to upgrade a version and rebuild a cloud host or restart a container. Monitoring objects change dynamically and very frequently. Even if fully automated, there will be adverse effects in terms of load and complexity.

Monitoring can also help with internal controls, especially in security-sensitive industries such as securities and banking. For example, when a server is attacked, we need to analyze the incident, find the root cause, identify similar attacks, discover unknown attacked systems, and even complete forensics.

The emergence of clusters makes the application topology complex, the metrics and log formats of different applications are not uniform, and the problem of dealing with multiple tenants has brought new challenges to monitoring.

Traditional monitoring includes monitoring of hosts, networks and applications, but after Docker appears, the container layer is easily ignored and becomes a blind spot for monitoring.

Some people say, isn't the container just an ordinary OS? Wouldn't it be enough to install a Zabbix probe? Docker hosts and Docker containers all need Zabbix probes…there are actually a lot of problems.

In addition to the CPU memory situation seen inside the container is not accurate, and the container life cycle is short, the host name and ip address will change after restarting, so it is best to install Zabbix agent on Docker host.

If every container were monitored like an OS, the metric count would be huge, and the data would likely be inefficient after a few minutes (the container would have stopped). The container life cycle is short, and once the container is finished, the previously collected data no longer has any meaning.

The main solution is to monitor in units of App or Service (through Tag, etc.).

##Docker Monitoring Technology Basics

docker stats

Remote API

pseudo file system

We can get container performance metrics through docker stats or Remote API and Linux pseudo-file system.

If you use the API, you need to be careful not to put a performance burden on the Docker daemon. If you have 200 containers per host, system performance can take up a lot of CPU time if you collect very frequently.

The best way is to use a pseudo-file system. If you just want to collect performance data via shell, docker stats is probably the easiest way to do it.

docker stats command

This command is streamed by default, and if you want to print out the latest data and exit immediately, you can use the no-stream=true parameter.

pseudo file system

CPU, Memory, Disk

network

The file location is probably (system-related, here's an example of Systemd):

Docker versions have different levels of support for these three methods, and the way to obtain metrics and the level of detail are also different, where network metrics are only available from pseudo-file systems after 1.6.1.

Memory

Many of the performance metrics for memory come from the memory.stat file:

The previous metrics without total represent the amount of memory used by the processes in the cgroup, excluding sub-cgroups, while the metrics at the beginning of total contain the data used by these processes, including sub-cgroups. Here we see the same data, since there are no child cgroups.

Two more important indicators:

RSS： resident set size

All data heaps, stacks, memory maps, etc. of the process. RSS can be further classified as active and inactive. When there is not enough memory to swap a part to disk, inactive rss will be selected for swap.

cache memory

Size of hard disk files cached in memory. For example, when you read or write files, or when you use mapped files, this memory will increase. This type of memory can also be subdivided into active and inactive caches, i.e. activefiles and inactive files. If the system needs more memory, inactive caches are reused first.

CPU

cpuacct.stat file

docker.cpu.system

docker.cpu.user

Unfortunately, Docker doesn't report nice, idle and iowait incidents.

System is also called kernel time, which is mainly the part consumed by system calls, while user refers to the CPU consumption of your own program. If User time is high, you need to check whether your program has problems and may need to be optimized.

Blkio

Priority is given to getting data from CFQ (Completely Fair Queuing), not from these two files: blkio.throttle.io servicebytes, number of bytes read and write· blkio.throttle.io_served, number of reads and writes

Throttle this simple may be misleading, in fact these are not limit values, but actual values. The first field of each file is a device ID in the format major:minor.

network data

iptables

pseudo file system

network device interface

Virtual Ethernet

Needle network monitoring should be accurate to the interface level, that is, the network card level. Each container has a corresponding virtual Ethernet on host, from which we can get tx and rx information.

However, it is troublesome to find the virtual network card corresponding to the container on the host. At this point, you can retrieve network data from inside the container using the ip netns command on the host.

In order to execute ip netns commands in the container's network namespace, we first need to find the PID of the container process.

Or:

In fact, Docker's implementation reads network metrics from pseudo-file systems:

At this point, I believe that everyone has a deeper understanding of "what is the principle of Docker data collection". Let's actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.