Shulou(Shulou.com)06/01 Report--

Phenomenon:

A high-equipped MySQL database server, dual CPU48 threads.

CentOS 7

The local disk is 15000 rpm Raid 0. System files, Swap are assigned to the local disk.

The server mounts an array of SSD disks. The database files are stored in the disk array.

After the system is running, it is found that the system Swap is widely used, and the IO utilization rate of the local disk lasts 100%.

Expedient measure:

I moved Swap to the disk.

Reason:

Swap insanity

NUMA architecture

Reference:

For single-CPU, multi-core cases, each core accesses memory at the same speed. This architecture is called SMP (Symmetric multiprocessing, symmetric multiprocessors), also known as UMA (Uniform Memory Architecture, as opposed to NUMA, consistent memory access architecture).

As you can see, each CPU has a set of matching memory slots. Each CPU accesses its own memory slot quickly, but for other memory slots on the motherboard, the access speed slows down. This architecture is called NUMA.

For Linux, memory is detected when loaded, calculating the CPU-to-memory access cost, and grouping CPU and memory into groups. Each process and thread inherits the NUMA policy of the parent process, which includes which CPU the process / thread will run on and which set of slots should be used for the allocated memory.

In the face of memory allocation, once allocated to the specified CPU- memory slot, it will not be moved. For applications such as databases, ideally it is a single multithreaded process that eats almost all the system memory and consumes as much of the rest of the system resources as possible, such as IO.

For the NUMA architecture of two CPU, there will be a problem if one core allocates more than half of the system memory. The allocation strategy for Linux is to use CPU 0 first and then CPU 1. At this point, there will be a situation where the memory group of CPU 0 has been used up first, but the system still has a lot of free memory, all on CPU 1. At this point, Linux chooses to brush the memory of CPU 0 to disk in exchange for available memory. However, the swap process is much slower than accessing memory across CPU. This will lead to the phenomenon that the memory has not been used up, but the database has been frantically flushed.

Reference:

Http://www.cnblogs.com/Lifehacker/p/database_swap_insanity_on_Linux.html

Https://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

Https://yq.aliyun.com/articles/50933

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.