Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to troubleshoot the lack of memory in the Linux system, and the content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Problem description

The Linux server memory usage exceeds the threshold, triggering an alarm.

Problem troubleshooting

First, observe the memory usage of the system through the free command, which is shown as follows:

Total used free shared buffers cached Mem: 24675796 24587144 88652 0 357012 1612488-/ + buffers/cache: 22617644 2058152 Swap: 2096472 108224 1988248

Here, you can see that the total memory is 24675796KB, 22617644KB has been used, and only 2058152KB is left.

Then, through the top command, shift + M sorts by memory, and then observes the processes that use memory * * in the system, and finds that only 18GB memory is occupied, and other processes are very small and can be ignored.

So, where else is the nearly 4GB memory (22617644KB-18GB, about 4GB) used?

Further, through cat / proc/meminfo, it is found that there is nearly 4GB (3688732 KB) of Slab memory:

. Mapped: 25212 kB Slab: 3688732 kB PageTables: 43524 kB.

Slab is used to store the kernel data structure cache, and then check the usage of this part of memory through the slabtop command:

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 13926348 13926348 / 0.21K 773686 18 3494744K dentry_cache 334040 262056 78% 0.09K 8351 40 33404K buffer_head 151040 150537 99% 0.74K 30208 5 120832K ext3_inode_cache

It is found that most of them (about 3.5GB) are used for dentry_cache.

Problem solving

1. Modify / proc/sys/vm/drop_caches to free up cache memory occupied by Slab (refer to the official documentation of drop_caches):

Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free. To free pagecache: * echo 1 > / proc/sys/vm/drop_caches To free dentries and inodes: * echo 2 > / proc/sys/vm/drop_caches To free pagecache, dentries and inodes: * echo 3 > / proc/sys/vm/drop_caches As this is a non-destructive operation, and dirty objects are notfreeable, the user should run "sync" first in order to make sure allcached objects are freed. This tunable was added in 2.6.16.

two。 Method 1 requires the user to have root permission. If it is not root, but has sudo permission, you can set it through the sysctl command:

$sync $sudo sysctl-w vm.drop_caches=3$ sudo sysctl-w vm.drop_caches=0 # recovery drop_caches

After operation, you can check whether it takes effect by using sudo sysctl-a | grep drop_caches.

3. Modify / proc/sys/vm/vfs_cache_pressure to adjust the priority of cleaning inode/dentry caches (default is 100). There is an explanation in LinuxInsight:

At the default value of vfs_cache_pressure = 100the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.

For specific setting methods, you can refer to either method 1 or method 2.

references

Https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Http://major.io/2008/12/03/reducing-inode-and-dentry-caches-to-keep-oom-killer-at-bay/

Http://linux-mm.org/Drop_Caches

The progress of further investigation is recorded below.

A deeper reason

The above troubleshooting shows that there is a large amount of dentry_cache in the Linux system taking up memory, why is there so much dentry_cache?

1. First of all, clarify the concept and function of dentry_cache: the directory item cache is designed by Linux to improve the processing efficiency of directory item objects; it records the mapping of directory items to inode. Therefore, when an application initiates a stat system call, a corresponding dentry_cache entry is created (further, if each stat file is a non-existent file, then a large number of new dentry_cache entries will always be created).

two。 The current server is a node of the storm cluster. The worker processes related to storm are first thought of. After strace, the worker process of storm finds that there are very frequent stat system calls, and the file of stat is always the new file name:

Sudo strace-fp-e trace=stat

3. Further observe that the worker process of storm frequently creates, opens, closes, and deletes heartbeat files in the local directory, with a new file name per second:

Sudo strace-fp-e trace=open,stat,close,unlink

This is why there are so many dentry_cache in the system.

A strange phenomenon

Through observation / proc/meminfo, it is found that slab memory is divided into two parts:

SReclaimable / / recyclable slabSUnreclaim / / non-recyclable slab

The current situation of the server at that time is that most of the memory occupied by slab is displayed as SReclaimable, that is, it can be recycled.

However, it is observed through slabtop that the OBJS of the most important part of slab memory (dentry_cache) is almost all ACTIVE, indicating that 100% is in use.

OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 13926348 13926348 100% 0.21K 773686 18 3494744K dentry_cache334040 262056 78% 0.09K 8351 40 33404K buffer_head151040 150537 99% 0.74K 30208 5 120832K ext3_inode_cache

Why is it recyclable but in the ACTIVE state? Ask the Linux kernel expert to explain enthusiastically after seeing it: (

Is it possible that the dcache is not automatically recycled and released because of the ACTIVE status?

Let the system automatically recycle dcache

As we mentioned in the previous section, most of the slab memory on the server is SReclaimable recyclable, so can we give it to the operating system to automatically trigger the recycling operation at some point? The answer is yes.

After checking some relevant information about Linux dcache, it is found that the operating system will not release the kswapd kernel process until it reaches the critical memory threshold. This threshold is calculated as follows:

1. First of all, grep low / proc/zoneinfo, the following results are obtained:

Low 1 low 380 low 12067

two。 Add up the above three columns and multiply by 4KB, this is the threshold. After calculating with this method, it is found that the current server recovery threshold is only 48MB, so it is difficult to see this phenomenon. In practice, it may not be possible to wait for recovery, and the operating system will be hang unresponsive.

3. You can increase this threshold by setting vm.extra_free_kbytes to the same size as vm.min_free_kbytes, the corresponding low threshold in / proc/zoneinfo will double, and the high threshold will increase accordingly, and so on.

$sudo sysctl-a | grep free_kbytes vm.min_free_kbytes = 39847vm.extra_free_kbytes = 0$ sudo sysctl-w vm.extra_free_kbytes=836787 # 1GB

4. For example, when the low threshold is set to 1GB, when the memory of the system free is less than 1GB, it is observed that the kswapd process starts to work (the process state changes from Sleeping to Running), and the dcache begins to be reclaimed by the system until the memory of the system free is between the low threshold and the high threshold.

On the Linux system memory shortage how to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.