Shulou(Shulou.com)06/01 Report--

We received a non-*, which is the Linux kernel version of 3.5-rc1 and the RedHat backport patch to deal with swappiness=0. This is a real threat, one of our customers was affected and the OOM mechanism was used to crash the MySQL master database server. This "minor" change to the kernel causes the system to fail to Swap properly, directly causing the OOM mechanism to kill the MySQL process. This raises doubts about the explanation that the system already has 128GB memory, a lot of memory is free, and has free virtual memory for 128GB, so the OOM mechanism should not be started under any circumstances.

We thought it was NUMA (we've written about NUMA before), but if that's the case, we'll see some excessive Swapping because of intra-node. We configured mysql-safe to use NUMA interaction mode by installing numctl, but eventually crashed.

It turns out that the server has a new kernel 2.6.32-358 of RHEL/Centos 6.4, which was released in February 2013. This version of the kernel and later versions have backport patches, and the system can be upgraded to 6.4 or higher, and we expect a lot of problems in this critical area.

This is frustrating because RedHat is not supposed to change some behaviors in backport or in a lifecycle like RHEL6, and their purpose is clear, and things like this won't happen, such as consistent behavior in the system's 5-10 years of life. So when a major problem like this arises in a product lifecycle, things are bad, such as mandatory upgrades, configuration changes, default installation upgrades, monitoring, and audit changes. Most of the latest Debian/Ubuntu systems will also have these problems because they also have updated kernels, perhaps the same backport.

Swappiness is often misunderstood by engineers. It can be set to a value from 0-100 to tell the kernel which is more important, pagecache (file cache) or application memory. The default value is 60, which means that more pagecache memory can be used, but this is a very misconfigured configuration for the server. From a virtualization perspective, all servers require application memory, more than file cache, so we always set it to 0, which means that file cache will be released until any application memory in swap. But now, this bug leads to less swapping, which greatly increases the chances that the OOM mechanism works under memory pressure, which is really not what we want. What is the technical solution that can be solved quickly? Fortunately, we have a very simple plan. Set swappiness to 1, which is almost the same priority as 0 to protect application memory, but does not trigger kernel changes. In this way, 1 to 0 is a better configuration.

As always, we will monitor and manage these types of issues for our customers, constantly upgrade the default installation configuration, and cycle through upgrades to affect the system.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.