Shulou(Shulou.com)06/03 Report--

The following figure shows several configuration requirements for the operating system environment in the official HBase documentation:

1. Close the transparent large page

Echo 'never' > / sys/kernel/mm/transparent_hugepage/enabled

Echo 'never' > / sys/kernel/mm/transparent_hugepage/defrag

/ / this step needs to be added to boot before it can be turned off.

/ / THP is a dynamic management strategy that allocates and manages large pages at run time, so there will be a certain degree of allocation delay, which is unacceptable for database systems that pursue response latency. When TPH is disabled, (never) HBase has the best performance and is relatively stable. On the other hand, the performance of the scene opened by THP (always) is about 30% lower than that of the closed scene, and the curve jitter is great. As you can see, remember to close THP on the HBase line.

2. Set vm.swappiness = 0 / / set to 0 here

/ / disable swapping (optional). Memory page swapping can cause CDH performance degradation in some cases.

[root@cloudera01 parcels513.el7] # echo vm.swappiness=1 > > / etc/sysctl.conf

[root@cloudera01 parcels513.el7] # sysctl-p

Vm.swappiness = 1

/ / the specific meaning is interpreted as:

Swappiness, this value is used to define how actively the kernel uses swap. The higher the value, the more active the kernel will use swap. The lower the value, the less active the kernel will use swap. The value ranges from 0 to 100, and the default is 60.

As far as the database is concerned, swap needs to be avoided as much as possible, so you need to set it to 1. It should be noted here that a setting of 1 does not mean that swap is not executed!

Tip:

The size of the value of swappiness has a lot to do with how swap partitions are used. Previously, it was suggested that vm.swapiness be set to 0, which means "do not swap memory unless there is a memory benefit." The meaning of this value did not change until the 3.5-rcl version of the Linux kernel was released. This change has been extended to other distributions, including the RedHat Enterprise Kernel 2.6.32-303. After the change, 0 means "do not exchange under any circumstances". So it is now recommended that this value be set to 1. Swappiness=100 means actively using swap partitions and moving the data in memory to swap space in a timely manner.

3. Set vm.min_free_kbytes to least 1GB (8GB on larger memory systems)

/ / this file represents the minimum amount of free memory (Kbytes) that is forced to be reserved by Linux VM. / / represents the minimum amount of free memory reserved by the system

The official document requires that the min_free_kbytes should not be less than 1 gigabyte (set 8 gigabytes in large memory systems), that is, do not easily trigger direct collection.

When the available memory is lower than this parameter, the system begins to reclaim cache memory to free up memory until the available memory is greater than this value.

Vm.min_free_kbytes=409600

Vm.vfs_cache_pressure=200

Vm.swappiness=40 .

The purpose of adjusting MIN_FREE_KBYTES is to keep enough free space in physical memory to prevent sudden page changes.

Swapiness defaults to 60, and reducing swapiness will allow the system to release more physical memory as soon as possible through process resources that are not used by swapout.

The default value of vfs_cache_pressure is 100. increasing this parameter sets the tendency for virtual memory to reclaim directory and i-node buffers. The higher this value, the more serious the tendency to recycle. The purpose of adjusting these three parameters is to allow the operating system to reclaim the buffer and release physical memory as soon as possible, so that sudden large-scale page changes can be avoided.

Https://blog.csdn.net/hanyingzhong/article/details/71637391?utm_source=blogxgwz0

The specific operations are as follows:

[root@NewCDH-0--141] # cat / proc/sys/vm/min_free_kbytes / / original

67584

[root@NewCDH-0--141 ~] # cat / proc/sys/vm/min_free_kbytes / / set to

1048576

[root@NewCDH-0--141 ~] # ll / proc/sys/vm/ | wc-l

forty-two

4. Disable numa zone reclaim with vm.zone_reclaim_mode = 0

/ / it is officially recommended that vm.zone_reclaim_mode = 0 be set to 0

[root@NewCDH-0--141 vm] # cat / proc/sys/vm/zone_reclaim_mode / / this parameter system defaults to 0

0

/ / what is UMA? What is the relationship between NUMA and swap? What is the specific meaning of zone_reclaim_mode?

Zone_reclaim_mode, this parameter defines different memory recovery policies under the NUMA architecture. You can take a value of 0 _ local _ 3 _ ax _ 4, where 0 means you can allocate memory to other memory areas if there is not enough local memory; 1 means that you can reclaim the memory locally before allocating it if there is not enough local memory; 3 means that the file cache objects are recycled locally as far as possible; 4 means that local recycling gives priority to using swap to reclaim anonymous memory.

What does swap do?

SWAP means swapping. As the name implies, when a process requests OS for insufficient memory, OS will swap out data that is temporarily unused in memory and put it in the SWAP partition. This process is called SWAP OUT. When a process needs this data again and OS finds that there is still free physical memory, it swaps the data from the SWAP partition back into physical memory, a process called SWAP IN.

Of course, there is a limit to the size of the swap, and once the swap is used up, the operating system will trigger the OOM-Killer mechanism to kill the process that consumes the most memory to free memory.

Obviously, the original intention of the swap mechanism is to alleviate the embarrassment of choosing a direct rough OOM process in order to run out of physical memory.

Why do database systems dislike swap?

1. Database systems are generally sensitive to response latency. If swap is used instead of memory, database service performance is bound to be unacceptable.

2. For systems that are extremely sensitive to response latency, there is no difference between too much delay and the service is unavailable. What is more serious than the unavailability of the service is that the process is immortal in the swap scenario, which means that the system has been unavailable. Then think about whether it is a better choice if you do not use swap to directly oom, so that many high-availability systems will directly switch between master and slave, and users are basically unaware of it.

3. In addition, for distributed systems such as HBase, they are not worried about the downfall of a node, but are just worried about the tamping of a node.

4. If a node goes down, at most, a small number of requests are temporarily unavailable and can be resumed after retry. However, a node tamping will tamp all distributed requests, server-side thread resources are occupied, resulting in the entire cluster request blocking, and even the cluster is dragged down.

The working Mechanism of swap

Since databases don't like swap, should we use the swapoff command to turn off the disk cache feature? Not also

Several requirements of the HBase official documentation actually implement this policy: to minimize the impact of swap. Only by knowing yourself and the enemy can you be invincible. In order to reduce the impact of swap, you must find out how Linux memory recovery works, so as not to leave out any possible doubts.

Let's take a look at how swap is triggered.

Linux triggers memory collection in two scenarios

1. one is that memory collection will be triggered immediately when it is found that there is not enough free memory during memory allocation.

2. One is to open a daemon (swapd process) to check the system memory periodically and actively trigger memory collection after the available memory is reduced to a certain threshold.

There are two main types of Linux memory recycling objects:

File caching, which is easy to understand, in order to prevent file data from being read from the hard disk every time, the system will store hot data in memory to improve performance. Anonymous memory, this part of the memory has no actual carrier, unlike the file cache has a hard disk file, such as typical heap, stack data and so on. This part of memory cannot be directly released or written back to a medium similar to a file when it is recycled, which leads to the swap mechanism, which swaps this kind of memory into the hard disk and loads it when needed.

Find https://mp.weixin.qq.com/s?__biz=MjM5NzAzMTY4NQ==&mid=2653929537&idx=1&sn=3fad622f505175d9ca8399cfb14b925f&chksm=bd3b5a2b8a4cd33d0bd75078614106ee4065b732684fb949d7477f45af56ae8be0a472fc75b1#rd, the culprit of SWAP on MySQL server.

Https://weibo.com/1680980510/EmvG57wVL?type=comment

There is still a lot of free physical memory, but it runs out of swap.

The vast majority of cases are caused by not shutting down NUMA. On the server running the database process, it is strongly recommended that you shut down NUMA

Solution:

Due to improper server hardware and system settings, NUMA was not turned off, resulting in SWAP. The recommended options are:

Turn off NUMA at the BIOS setting level, but the disadvantage is that you need to restart OS

Or modify the GRUB configuration file, and the disadvantage is to restart OS.

Upgrade MySQL version to 5.6.27 and later, a new option innodb_numa_interleave has been added. You only need to restart the mysqld instance, and there is no need to restart OS. This solution is recommended.

The reference link mentioned above in the Linux initialization script (centos6 centos7 generic) https://blog.51cto.com/12445535/2362407 is: http://hbasefly.com/2017/05/24/hbase-linux/

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.