Shulou(Shulou.com)05/31 Report--

This article mainly explains what redis-cluster needs to pay attention to in Redis. Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn what redis-cluster needs to pay attention to in Redis.

1. 150 alarm received. Rdb persistence failed.

15011Sep M 17 Sep 08 Cannot allocate memory15011:M 54 Sep 037 # Can't save in background: Cannot allocate memory15011:M 17 043 * 1 changes in 900 seconds. Saving...15011:M 17 Sep 08:54:49.043 # Can't save in background: fork: Cannot allocate memory

2 check the host memory (inner os: Nima still has so much memory)

[root@ip-172-31-43-150] # free-g total used free shared buff/cache availableMem: 29 14 100 4 14Swap: 0 000

3 check the status of the redis-cluster cluster and show that 150 has been downloaded and flustered

[root@ip-172-31-39-42] # / usr/local/src/redis-4.0.8/src/redis-trib.rb check 172.31.39.42 ERR [ERR] Sorry Can't connect to node 172.31.43.150 Performing Cluster Check 6379 claims to be slave of unknown node ID 6d2b67b9745a8d4bedb70d480645e3651fddaf3f. > Performing Cluster Check (using node 172.31.39.42 using node 6379) M: 00f7bd511046438af2d1b41666a69ff77b6f176f 172.31.39.42 using node 6379 slots:11258-11832 slots) master 1 additional replica (s) S: e771e70f580ec2799af50268865444cf425e000e 172.31.33.1779 slots: (0 slots) slave replicates 00f7bd511046438af2d1b41666a69ff77b6f176fS: 8bb99c5b9585269b66684400f036fca1d30e72cb 172.31.47. 157slots:28,4799 6379 slots: (0 slots) slave replicates 148697f75e9b4f84ad893f4d5377e96fdde7664dM: 148697f75e9b4f84ad893f4d5377e96fdde7664d 172.31.34.25 slots:28,4799-5462 master 6375-7282 slots: (0 slots) slave replicates 792ab7473fa447d07582817eb2f489633001d831M: 792ab7473fa447d07582817eb2f489633001d831 172.31.36.10 slots:28,4799: (0 slots) slave replicates 792ab7473fa447d07582817eb2f489633001d831M: 792ab7473fa447d07582817eb2f489633001d831 172.31.36.10 master 1 additional replica (s) Slots:0-27 slave replicates 6d2b67b9745a8d4bedb70d480645e3651fddaf3fS 29-1145Power1822-2105 additional replica 3406-7541 (3081 slots) master 1 additional replica (s) S: 92a5541964fc3e4bfb90f1750b9105d5705beb93 172.31.39.54 additional replica 6379 slots: (0 slots) slave replicates 6d2b67b9745a8d4bedb70d480645e3651fddaf3fS: 7e5e1e341f33ebd7a3c20480b66a76bbd0922a4f 172.31.2546379 slots: (0 slots) slave replicates 40b766b505c54066de5b5d8eb214ea78c7df8c4b [OK] All nodes agree about slots configuration. > > Check for open slots... > Check slots coverage... [ERR] Not all 16384 slots are covered by nodes.

Check the status of redis and find that it is fine!

First solve the problem of persistence failure:

1.172.31.39.54 config rewriteOK172.31.39.54:6379 6379 > config set stop-writes-on-bgsave-error no-solve the problem of throwing exceptions on the application side OK172.31.39.54:6379 > config rewriteOK172.31.39.54:6379 > 2. Enable kernel parameters to solve the problem of bgsave failure [root@ip-172-31-33-182] # sudo echo 'vm.overcommit_memory = 1' > > / etc/ sysctl.confession [root @ ip-172-31-33-182 ~] # sysctl-pvm.overcommit_memory = 1

Check the log again, it has been persisted successfully, and the check cluster also found that the cluster returned to normal.

Learn about memory allocation for redis:

Redis has its own memory allocator, and when a key-value object is removed, Redis does not immediately release its occupied memory to the operating system (for example, when a user populates an instance with 5G of data and removes 2G of data, but the occupied memory may remain around 5G). Why would Redis deal with it like this? There are two reasons: 1. OS may swap free memory to VM, but the VM of OS is a physical file, and its IO read and write efficiency is low, thus affecting Redis performance; 2. OS's VM swap is based on the Page mechanism, and some data objects in the same Page are released, but other data objects are still in use by other applications, resulting in the Redis objects in the Page are not released. The Redis author should consider the above problems and do not want Redis to degrade performance, so Redis prefers to control the granularity of VM swapping in design. (https://segmentfault.com/a/1190000004708270)

Persistence problem

Redis persistence disk IO mode and its problems people with Redis online operation and maintenance experience will find that Redis uses more physical memory, but the problem of instability or even collapse occurs when it does not exceed the actual total physical memory capacity. Some people think that it is caused by the double memory consumption caused by fork system calls persisted based on snapshots, this view is not accurate. Because the copy-on-write mechanism of fork calls is based on the unit of operating system pages, that is, only dirty pages that have been written will be copied, but generally your system will not have all pages written in a short period of time and cause replication, so what causes Redis to crash? The answer is that the persistence of Redis is caused by the use of Buffer IO. The so-called Buffer IO refers to the Page Cache in which Redis uses physical memory to write and read persistent files, and most database systems use Direct IO to bypass this layer of Page Cache and maintain a data Cache on their own, while when the persistent files of Redis are too large (especially snapshot files) and read and write them The data in the disk file will be loaded into the physical memory as a layer of Cache for the file by the operating system, and the data of this layer of Cache and the data managed in Redis memory are actually stored repeatedly. Although the kernel will do Page Cache elimination when the physical memory is tight, the kernel will probably think that a certain Page Cache is more important and let your process start Swap, and your system will begin to become unstable or crash. Our experience is that it starts to be dangerous when your Redis physical memory usage exceeds 3 / 5 of the total memory capacity. At this point, I believe you have a deeper understanding of "what redis-cluster needs to pay attention to in Redis". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.