Shulou(Shulou.com)06/01 Report--

This article introduces the Linux actual memory occupancy algorithm and how to use Python to achieve memory monitoring, the content is very detailed, interested friends can refer to, hope to be helpful to you.

In the past two days, one of our core systems, a set of clusters, began to alarm one by one, with the content of memory occupation exceeding the threshold. This is supposed to be a very urgent alarm that needs to be dealt with immediately, but whether this is really the case will wait for us to see the sun.

The following is a simulation experiment

Understand the next monitoring system this memory occupancy over-threshold alarm calculation method, is the use of used/total*100%=1949/1988*100%=98%, that is, memory consumption is more than 98%.

Online learning, in fact, this calculation method is not very accurate, the reason is that the memory management mechanism of Linux and Windows is not the same, I understand the characteristics of Linux memory management, one of them is to make full use of memory, online information is very easy to retrieve, I will not learn how to play tricks. I have seen some articles on MOS, which directly or indirectly illustrate this problem.

(1) Memory Not Being Released In Linux Top After The MDEX Engine Is Shut Down (document ID 2045288.1)

The question is, why did you shut down the software and did not see the free memory of the application in top? His explanation is that this is normal, and that the RAM shown in the memory part of the top instruction is the "cache" part. These cached portions represent the most recently used data on disk and are temporarily stored in RAM. When an application is stopped, memory is not "freed". Once the application is restarted, this information is in the cache and can improve performance. However, if the application is closed for a long time and other processes need this memory, it can be used seamlessly. The operating system reclaims as few segments of data as possible and allocates them to new processes.

(2) Summary Area in top output Shows Low Free Memory (document ID 460141.1)

The Unix system has been optimized to make full use of all available resources. Memory resources that are not used by code or data can be used by OS for a variety of purposes. The summary part of the output of top instructions does not describe the memory of the kernel and user processes, and the amount of free memory is very low, which does not explain the problem of memory usage. Unused memory is used by the operating system to cache data that has recently been accessed and will be used again in the near future, but it can be easily freed if some of the more important processes need it.

So how should the real memory usage be calculated?

First of all, let's talk about the difference between buffers and cached in free output. Reference (description of the difference between buffers and cached given by http://blog.chinaunix.net/uid-24709751-id-3564801.html)

(a) buffers refers to the size of the buffer used for block devices. It records only the metadata and tracking in-flight pages of the file system. Cached is used to buffer files.

(B) buffers is used to store what is in the directory, permissions, etc. While cached is directly used to remember the files we open, such as executing two commands # man X one after another, you can clearly feel that the second start is much faster. While buffers is increasing all the time, for example, after using ls / dev twice, you will find that the second execution will be faster than the first.

There are different understandings of memory usage from different angles. The following figure is an example.

(1) from the operating system's point of view, the memory of buffer and cache belongs to the allocated memory, so the memory occupancy calculation method = used/total*100%.

(2) from the user's point of view, the memory of buffer and cache can be reused, so the memory occupancy calculation method = (used-buffers-cached) / total*100%.

So for the initial question, we focus on the application available memory, so we need to monitor the actual memory occupancy of the concern should be

(used-buffers-cached) / total*100%= (1949-99-1053) / 1988 / 100% / 40%

Furthermore, there are different ways to monitor memory usage, no matter from online information or actual work experience. Here I just briefly talk about how to use Python to monitor memory usage.

(1) implement without using any third-party libraries

The idea is very simple, that is, execute the free instruction, parse the echo, get the parameter values of used, buffers, cached, total, and then calculate them.

Output = os.popen ('free-m')

Output.readline ()

Line =', '.join (output.readline (). Split ()) .split (',')

Total = float (line [1])

Used = float (line [2])

Buffers = float (line [5])

Cached = float (line [6])

Percent = int (round ((used-buffers-cached) / total * 100))

The result is rounded up, for example, 25%. Note that you need to use the float floating point type, otherwise it will be 0 before multiplying the percent by 100.

(2) use the third-party library psutil

Psutil is a cross-platform process management, which needs to be installed first. Before psutil installation, python-devel needs to be installed, both of which require root users. My operating system is Linux 6.5 Magi Python version 2.6.6, and the versions of these two packages are

(a) python-devel-2.6.6-51.el6.x86_64.rpm

(B) psutil-2.0.0

Using a third-party library, this problem is very easy.

Import psutil

Percent = mem.percent

Mem.percent= (mem.total-mem.available) / mem.total * 100 here, where mem.available=mem.free + mem.buffers + mem.cached is consistent with the calculation method in (1).

In fact, psutil can perform almost all instructions and operations related to system monitoring.

Github address, https://github.com/giampaolo/psutil/

The introduction to this article is very clear, http://www.jianshu.com/p/64e265f663f6

Psutil (Python system and process utilities) is a cross-platform python library for process management and system tools, which can deal with system CPU,memory,disks,network and other information. It is mainly used for the monitoring, analysis and management of system resources. Through psutil, we can realize such as ps,top,lsof,netstat,ifconfig, who,df,kill,free,nice,ionice,iostat,iotop,uptime,pidof,tty,taskset,pmap. Working on systems such as Linux,windows,OSX,freebsdSun Solaris, the latest version of python is higher than 2.6 (Python 2.4 Python2.5 can use version 2.1.3)

Summary:

1. What we are actually concerned about under Linux is the utilization rate of the available memory of the system. The calculation method of (used-buffers-cached) / total*100%, is not the part shown by free in the execution of free instructions.

two。 Memory usage monitoring Python implementation, you can use the execution of free instructions to parse the output, if you can use psutil third-party libraries, it is even easier.

About the Linux actual memory occupancy algorithm and how to use Python to achieve memory monitoring is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.