Shulou(Shulou.com)05/31 Report--

This article is about how to troubleshoot the weird growth of Redis memory. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

I. phenomenon

Instance name: r-bp1cxxxxxxxxxd04 (master-slave)

Problem: memory has increased by 2G in one minute, as shown in the following figure:

Key size: about 60 million

Memory increases by one minute 2G.png

2. Redis memory analysis

1. Memory composition

The memory in the figure above counts the used_memory attribute in the info memory command of Redis, for example:

Redis > infomemory#Memoryused_memory:9195978072used_memory_human:8.56Gused_memory_rss:9358786560used_memory_peak:10190212744used_memory_peak_human:9.49Gused_memory_lua:38912mem_fragmentation_ratio:1.02mem_allocator:jemalloc-3.6.0

Detailed description of each attribute

The attribute name attribute indicates the amount of memory allocated by the used_memoryRedis allocator, that is, the total amount of memory actually stored in the data used_memory_human returns the total amount of memory used by Redis in a readable format used_memory_rss from the operating system's point of view, the total physical memory occupied by the Redis process, the maximum memory allocated by the used_memory_peak memory allocator Used_memory_peak_human, which represents the historical peak of used_memory, displays the mem_fragmentation_ratioused_memory_rss / used_memory ratio of memory consumed by the peak memory consumption used_memory_luaLua engine in a readable format, indicating the memory allocator used by mem_allocatorRedis. Default: jemalloc

The calculation formula is as follows:

Used_memory = own memory + object memory + buffer memory + lua memory used_rss = used_memory + memory fragmentation

As shown in the following figure:

two。 Memory analysis

(1) self-memory: an empty Redis occupies a small amount and can be ignored.

(2) kv memory: key object + value object

(3) buffer: client buffer (normal + slave camouflage + pubsub) and aof buffer (relatively fixed, generally no problem)

(4) memory consumed by Lua:Lua engine

3. Common problems with memory bursts

(1) kv memory: bigkey, bulk write

(2) client buffer: common client buffer (such as monitor command) or pubsub client buffer

III. Troubleshooting

(1) bigkey? No bigkey was found after scanning.

Sampled 67234427 keys in the keyspaceful Total key length in bytes is 1574032382 (avg len 23.41) Biggest string found 'CCARD_DEVICE_CARD_REF_MAP_KEY_016817000004209' has 20862 bytesBiggest list found' CCARD_VALID_DEVICE_TRAIN_QUEUE_KEY' has 51 itemsBiggest hash found 'CCARD_VALID_DEVICE_TRAIN_MAP_KEY' has 51 fields67234359 strings with 71767890 bytes (100.00% of keys, avg size 1.07) 67 lists with 151items (00.00% of keys, avg size 2.25) 0 sets with 0 members (00.00% of keys) Avg size 0.00) 1 hashs with 51 fields (00.000% of keys, avg size 51.00) 0 zsets with 0 members (00.005% of keys, avg size 0.00)

(2) the number of key values increases? No obvious change in key value was found.

(3) client buffer

Since the memory has not fallen for a long time after the memory has been increased, if it is due to a buffer problem, an obvious problem will be found in info clients. After execution, it will be found:

Redis > info clients# Clientsconnected_clients:43client_longest_output_list:0client_biggest_input_buf:0blocked_clients:0admin_clients:6rejected_vpc_conn_count:0close_idle_unknown_conn_count:0

There is no obvious case of omem greater than 0 in the execution of client.

Id=80207addr=10.xx.0.4:63920fd=46name=age=624idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80215addr=10.xx.0.23:43489fd=36name=age=591idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80366addr=10.xx.0.8:59785fd=18name=age=84idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=delread=0write=0type=user

Id=80356addr=10.xx.0.33:32117fd=13name=age=114idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80064addr=10.xx.59.4:53446fd=38name=age=1070idle=1070flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin

Id=80276addr=10.xx.0.23:48511fd=8name=age=387idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80188addr=10.xx.0.33:16265fd=42name=age=681idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80326addr=10.xx.0.32:59779fd=16name=age=209idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80065addr=10.xx.59.4:53447fd=45name=age=1070idle=1070flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin

Id=79936addr=10.xx.0.22:10607fd=30name=age=1480idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80174addr=10.xx.0.5:60914fd=6name=age=722idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80300addr=10.xx.0.22:22757fd=48name=age=298idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80037addr=10.xx.0.5:55189fd=15name=age=1143idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80330addr=10.xx.0.8:48533fd=17name=age=199idle=10flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=79896addr=10.xx.0.30:26814fd=11name=age=1616idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80299addr=10.xx.0.24:11227fd=44name=age=303idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80086addr=10.xx.0.32:52526fd=40name=age=1002idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80202addr=10.xx.0.33:16658fd=26name=age=636idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80256addr=10.xx.0.24:60496fd=19name=age=448idle=2flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=79908addr=10.xx.0.29:18975fd=12name=age=1583idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80365addr=10.xx.0.29:46429fd=14name=age=85idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=79869addr=10.xx.27.4:48455fd=35name=age=1700idle=1700flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin

Id=80334addr=10.xx.0.23:50012fd=39name=age=189idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80041addr=10.xx.0.32:51107fd=33name=age=1132idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=79992addr=10.xx.0.22:12068fd=28name=age=1289idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80251addr=10.xx.0.30:44213fd=23name=age=468idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80006addr=10.xx.0.2:45895fd=31name=age=1242idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80321addr=10.xx.0.30:48048fd=5name=age=224idle=3flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80381addr=10.xx.0.8:13360fd=22name=age=24idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=delread=0write=0type=user

Id=80200addr=10.xx.0.24:59183fd=24name=age=640idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80113addr=10.xx.0.2:52492fd=21name=age=915idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=174addr=11.216.117.242:53027fd=9name=age=281390idle=0flags=S db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=replconf read=0write=0type=admin

Id=79991addr=10.xx.0.4:48412fd=25name=age=1296idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80301addr=127.0.0.1:47869fd=49name=age=291idle=261flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=strlen read=0write=0type=admin

Id=80047addr=10.xx.59.4:53184fd=41name=age=1114idle=1114flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin

Id=80236addr=10.xx.0.5:62546fd=47name=age=516idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80364addr=10.xx.0.4:18794fd=7name=age=85idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80175addr=10.xx.0.4:62245fd=29name=age=718idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80336addr=10.xx.0.29:45701fd=50name=age=180idle=1flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80050addr=10.xx.59.4:53188fd=43name=age=1114idle=1114flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=NULL read=0write=0type=admin

Id=79765addr=10.xx.0.2:33832fd=37name=age=2027idle=177flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=info read=0write=0type=user

Id=80170addr=10.xx.0.2:57853fd=20name=age=728idle=24flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=0obl=0oll=0omem=0events=r cmd=ping read=0write=0type=user

Id=80390addr=127.0.0.1:49449fd=27name=age=0idle=0flags=N db=0sub=0psub=0multi=-1qbuf=0qbuf-free=32768obl=0oll=0omem=0events=r cmd=client read=0write=0type=admin

IV. Find out the culprit

Several commonly used tricks have been used, but it still doesn't work. My colleague @ Jingyuan helps to analyze it. I wonder if it is because Redis's kv hash table has done rehash.

1. Kv storage structure of Redis

As shown in the following figure, all the kv of Redis is stored in dict, where ht corresponds to two hash tables ht [0] and ht [1]. Usually, one is free, one is used to store data, and ht [1] is used only when rehash is needed.

2. Redis's dictionary rehash

In order to ensure the load of the hash table, rehash expansion will be carried out when the number of elements of the hash table is equal to the number of hash table slots. The capacity of h [1] after expansion is equal to the first 2n greater than or equal to ht [0] .size * 2. For example, the initialization capacity of hash table is 4, then the next expansion is 8, and so on.

3. test

(1) Test method

First write in batches near the rehash threshold, and then write one by one to observe the memory changes

/ / set the 1-day expiration time for each key int expireTime = 60 * 60 * 24 rehash threshold-50 to facilitate the observation of rehash memory changes int rehashThreshold = (int) Math.pow (2,25)-50 investors / 1. Batch writes: pipeline batch writes. Since it is a local test, 10000 is used here. Do not use Pipeline pipeline = jedis.pipelined (); pipeline = jedis.pipelined (); for (int I = 0; I < rehashThreshold; iTunes +) {pipeline.setex (String.valueOf (I), expireTime, String.valueOf (I)); if (I% 10000 = = 0) {pipeline.sync ();}} pipeline.sync (); / / 2. Wait to write incremental TimeUnit.SECONDS.sleep (5); for (int I = rehashThreshold; I < rehashThreshold + 200; iTunes +) {jedis.setex (String.valueOf (I), expireTime, String.valueOf (I)); TimeUnit.SECONDS.sleep (1);}

(2) start testing

(a) when the threshold = 21532768, it can be seen from below that the number of key is 32769, the memory increases a little, but it is not obvious.

Keys mem clients blocked requests connections32766 4.69M 3 0 32797 (+ 2) 4

32767 4.69M 3 0 32799 (+ 2) 4

32768 4.69M 3 0 32801 (+ 2) 4

32769 5.44M 3 0 32803 (+ 2) 4

(B) when the threshold = 220 1048576, it can be seen from the following that the number of key is 1048577, the memory increases by 32m. Because rehash expands, the slot in the new hash table becomes 221x2 (because each key has an expiration time set for the expires table), with a pointer of 8 bytes, 221? 2? 8 = 225 = 32MB.

Keys mem clients blocked requests connections1048574 128.69M 3 0 3364129 (+ 2) 16

1048575 128.69M 3 0 3364131 (+ 2) 16

1048576 128.69M 3 0 3364133 (+ 2) 16

1048577 160.69M 3 0 3364135 (+ 2) 16

1048578 160.69M 3 0 3364137 (+ 2) 16

(C) when the threshold value = 2260108864, it can be seen from below that the number of key is 67108865, the memory increases 2GB. Because rehash expands, the slot in the new hash table becomes 227x2 (because each key sets the expiration time, the expires table), the pointer is 8 bytes, 2272? 8 = 231 = 2GB.

Keys mem clients blocked requests connections67108862 9.70G 3 0 70473683 (+ 2) 18

67108863 9.70G 3 0 70473685 (+ 2) 18

67108864 9.70G 3 0 70473687 (+ 2) 18

67108865 11.70G 3 0 70473689 (+ 2) 18

67108866 11.70G 3 0 70473691 (+ 2) 18

67108867 11.70G 3 0 70473693 (+ 2) 18

Looking back at the key and memory change diagram of r-bp1c15fd9b142d04, you can see that the above rules are correct:

4. Follow-up observation

At 17:00, the rehash ended and the memory was halved by the increased 2G.

Thank you for reading! This is the end of the article on "how to troubleshoot the weird growth of Redis memory". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.