Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how Container internal process monitoring is". In daily operation, I believe many people have doubts about what Container internal process monitoring is. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "Container internal process monitoring". Next, please follow the editor to study!

At present, there are many kinds of virtualization technologies in the market, such as moby (docker), LXC, RKT and so on. While bringing the benefits of convenient application deployment and full use of resources, how to monitor the corresponding Container and its internal application process has become a new situation that operators will inevitably encounter. Starting from the basic principles of virtualization technology and the kernel characteristics of Linux operating system, UAV.Container obtains the monitoring data of various dimensions of Container containers and internal processes, so that appropriate monitoring dimensions can be obtained from the point of view of virtual machines, physical machine operation and maintenance personnel, or business operation and maintenance personnel.

From the basic principle, virtualization technology is mainly the application of cgroups, namespace and file system, and the operating system, as the root node of cgroup and namespace, no matter what kind of application is started in container, from the kernel point of view, the operating system must have its certain characteristics and forms of expression. What we need to do is to process these features in order to get the corresponding monitoring data.

Let's take docker technology as an example, and other virtualization technologies are similar.

1. Container ID

Container ID is the unique identity of a Container. From the perspective of container monitoring, we need to be able to find out which Container the process is running in. At the operating system level, the mount of the process's cgroup can be reflected. As shown in the figure, we run a Tomcat process inside a Container where ID is 3411554ff684.

Since the pid namespace of Container is a child namespace of the pid namespace of the operating system, the process should also have a corresponding pid at the operating system level. Verify it with the docker top command:

The process number of the process in the container on the host is 1848. Next, go to / proc/1848/cgroup to take a look at the cgroup mount of the process

The virtualization technology that implements the container, the Container ID and the resource mount path of the container are clearly shown in the cgroup file. Compare the Container ID shown here, which is exactly the same as the ID when the Container was created. This also verifies that Container ID can be obtained by scanning the cgroup information of the host process. This associates pid with Container ID.

II. CPU

Although cgroup controls the CPU usage of all processes under the cgroup, from the operating system's point of view, no matter whether the process belongs to a sub-cgroup or not, it is still the CPU of the shared host. So monitoring the CPU of the process on the host machine can get the CPU monitoring index of the process.

The common CPU monitoring command on Linux is top. The principle of top monitoring CPU is to obtain the cumulative total time countAll1 and busy total time countBusy1 of CPU from startup at the time of time1, and then obtain the total time countAll2 of CPU and total time countBusy2 of busy at the time of time2. Finally, the occupation of machine CPU in the time period from time1 to time2 is obtained by subtracting the difference of total time from busy. That is:

CPU occupancy (%) = (countBusy2-countBusy1) / (countAll2-countAll1) * 100

Similarly, if you get the total busy time countProcBusy1 and countProcBusy2 of each process at two times, you will get the CPU utilization rate of the process:

Process CPU occupancy (%) = (countProcBusy2-countProcBusy1) / (countProcAll2-countProcAll1) * 100

The total CPU time slice of the host from startup can be obtained from / proc/stat:

The first line is the overall CPU usage, the meaning of the specific parameters:

Therefore, time2,countAll = user + nice + system + idle + iowait + irq + softirq + stealstolean + guest + guest_nice is selected after the current time1,3 seconds. CountBusy is the value of countAll minus idle, so that all the required values of the first formula above are complete and can be calculated directly.

The second and third lines are the usage of each logical CPU. Note here that the logical cores of two logical CPU,CPU are related to the CPU display modes irix and solaris.

Next is the calculation of countProcBusy. The CPU time slice of the process is located under / proc/$pid/stat, as shown in the figure:

This file contains a lot of information about the process. Among them, 14, 15, 16 and 17 parameters are related to CPU.

So, countProcBusy = utime + stime + cutime + cstime, which includes the cpu time of all its threads. And countProcAll2-countProcAll1=3s, through the two moments of countProcBusy and countProcAll, the CPU utilization of the process can be obtained.

There are two points to be paid attention to:

1) jiffies actually refers to the total number of beats used by the kernel clock, so the jiffies here needs to be converted into seconds to apply the division formula above.

2) just now we talked about CPU display modes irix and solaris. To put it simply, irix mode means that the machine has N logical CPU,CPU display limits, that is, the upper limit of Number100% CPU,CPU display is 100%, no matter how much logical CPU,CPU display the upper limit is 100%, and / proc/$pid/stat shows that all logical CPU times are calculated, so the two display modes mean that the calculation methods are slightly different. The result of the solaris mode needs to be divided by the logical core number based on the above process CPU occupancy formula.

III. Memory

The monitoring of process memory has two dimensions of data: one is the physical memory footprint, the other is the percentage of process memory footprint.

Process memory occupancy (%) = process physical memory footprint / total host memory size * 100

Similar to CPU, the / proc/$pid/status file records the physical memory usage of the process, where VmRSS is the actual physical memory currently occupied by the process.

The memory usage of the machine is recorded under the / proc/meminfo file. This file is very long. Capture a part of it to show that MemTotal is the total memory size of the host:

In this way, the physical memory footprint of the process and the total machine memory are obtained, and the corresponding process memory occupancy is also obtained.

4. Disk IO

It is also very easy to get disk IO. / proc/$pid/io has helped us record the io of this process, but similar to CPU, the io file also stores the total io of the process from startup to the present, so:

Disk I time2 O (bytes/ seconds) = (time2 moment I time1 O) / (time2-time1)

The read_bytes and write_bytes are the number of bytes read and written from the start of the process, respectively, and take the values of two times respectively. According to the above formula, the disk IO of the process is obtained.

Port number and number of connections

Because the Network Namespace isolates the network, if the process runs inside the Container, the port information of the process should also be the port number that the process itself listens to, rather than the actual external port, while the mapping mechanism of the internal and external ports of the Container is controlled by the virtualization technology of the application itself, so it is inevitable to deal with the virtualization technology of the container. The problem then translates into getting the port that the process itself listens to in the container.

/ proc/$pid/net/tcp (tcp6,udp,udp6) keeps a history of port numbers and the number of connections. These file formats are similar, taking tcp6 as an example

Explain several key key:

Because st = 0A represents listen, the data with st = 0A is selected, and the corresponding inode number is extracted. Here the inode number is socket number, and the question is whether the socket represented by this socket number still exists in this process. With all the fd (file descriptor) of the process under / proc/$pid/fd, intercept a paragraph for example.

The soft chain after each file descriptor is actually the open file, starting with socket is the socket opened by the process, and the part in the middle of the brackets is the socket number. Make a match between the socket number and the inode number obtained in the above tcp6. If it matches, then the port st = 0A in the tcp6 is monitored by this process. As for the mapping of ports inside and outside the container, this needs to be obtained according to the mapping method of the virtualization technology applied. The calculation of the number of connections is the same as port scanning, except that st = 01 (establish) needs to be scanned and counted.

At this point, the study of "what is the internal process monitoring of Container" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.