Shulou(Shulou.com)06/02 Report--

This article focuses on "how to clean up useless data in Redis". Friends who are interested may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to clean up useless data in Redis.

In our data platform, we use Redis to provide high concurrent read and write requests with low latency (less than 20 milliseconds) online. * Redis uses Aliyun's Redis cluster (256g), which stores more than 1 billion records. The validity period of Key is set to 15 days. About 50 million records are written every day, and QPS is about 60,000. Because the generation speed of expired Key is faster than the speed of automatic cleaning of Redis, there will be a large number of expired Key in Redis that are not cleaned in time.

Why are there expired Key not cleaned up? You need to familiarize yourself with Redis's deletion strategy first.

There are three common deletion strategies for Redis:

Passive deletion (lazy deletion): when reading / writing an expired Key, the lazy delete policy is triggered and the Key is deleted directly

Proactive deletion (regular deletion): Redis conducts regular inspections to clean up expired Key

When the memory reaches the maxmemory configuration, the deletion of Key will be triggered.

In addition, there is a trigger-based deletion strategy, which is generally unused because of the pressure on Redis.

Here are the last two deletion strategies (there are a lot of instructions online).

Proactive deletion (periodic deletion)

In Redis, general operations are implemented by redis.c/serverCron, which mainly performs the following operations:

Update all kinds of statistics of the server, such as time, memory footprint, database footprint, etc.

Clean up expired key-value pairs in the database.

Resize unreasonable databases.

Close and clean up clients with failed connections.

Try an AOF or RDB persistence operation.

If the server is the primary node, periodically synchronize the subsidiary nodes.

If you are in cluster mode, perform regular synchronization and connectivity tests on the cluster.

Redis runs serverCron as a time event, ensuring that it runs automatically at regular intervals, and because serverCron needs to run periodically while the Redis server is running, it is a cyclical time event: serverCron executes periodically until the server shuts down.

In Redis 2.6, the program requires serverCron to run 10 times per second, an average of once every 100ms. Starting with Redis 2.8, users can adjust the number of serverCron execution per second by modifying the hz option. For more information, please refer to the description of the hz option in the redis.conf file.

Also known as scheduled deletion, the term "periodic" here refers to the cleaning policy that is periodically triggered by Redis, which is done by the activeExpireCycle (void) function located in src/redis.c.

ServerCron is a positioning task driven by redis's event framework. In this scheduled task, the activeExpireCycle function is called to delete expired key for each db within the limited time REDIS_EXPIRELOOKUPS_TIME_LIMIT. The reason for limiting the time limit is to prevent long-term blocking from affecting the normal operation of the redis. This active deletion strategy makes up for the memory unfriendliness of the passive deletion strategy.

Therefore, Redis will periodically randomly test a batch of key with expiration time set and process them. Expired key tested will be deleted. Typically, Redis does the following steps 10 times per second:

Randomly test 100 key with expiration time set

Delete all found expired key

Repeat step 1 if more than 25 key are deleted

This is a simple algorithm based on probability. The basic assumption is that the extracted samples can represent the entire key space, and redis continues to clean up expired data until the percentage of key that is about to expire is reduced to less than 25%. This also means that the maximum amount of key that has expired but still occupies memory space at any given time is the number of writes per second divided by 4. 5%.

The default value in Redis-3.0.0 is 10, which means that background tasks are called 10 times per second.

In addition to the frequency of active elimination, Redis also has a limit on the duration of each elimination task, which ensures that there will not be too many blocking application requests each time it is actively eliminated. Here is the limited calculation formula:

# define ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC 25 / * CPU max% for keys collection * / … Timelimit = 1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100

Scaling up hz will increase the frequency of active elimination of Redis. If your Redis storage contains a lot of cold data that takes up too much memory, you can consider increasing this value, but the Redis authors recommend that this value not exceed 100. We actually increased this value to 100 online and observed that CPU increased by about 2%, but there was a significant increase in memory release speed for cold data (by observing the number of keyspace and used_memory size).

You can see that the relationship between timelimit and server.hz is reciprocal, that is, the larger the hz configuration, the smaller the timelimit. In other words, the higher the expected frequency of active elimination per second, the shorter the maximum duration of each phase-out. Here, the maximum phase-out time per second is a fixed 250ms (1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/100), while the phase-out frequency and the maximum time of each phase-out are controlled by the hz parameter.

From the above analysis, when the expired key ratio in the redis does not exceed 25%, increasing the hz can significantly increase the minimum number of scanned key. Assuming that hz is 10, at least 200 key are scanned in a second (10 calls per second * at least 20 key are randomly selected each time). If hz is changed to 100, then at least 2000 key; are scanned in one second. On the other hand, if the expired key ratio exceeds 25%, there is no limit to the number of scanned key, but the cpu time takes up 250ms at most per second.

When REDIS is running in master-slave mode, only the master node executes these two expired deletion policies, and then synchronizes the deletion operation "del key" to the slave node.

Maxmemory

When the current used memory exceeds the maxmemory limit, trigger active cleanup policies, which can be configured (parameter maxmemory-policy), including the following:

Volatile-lru: select the least recently used data elimination from the dataset with an expiration time set (server. DB [I]. Obsolete).

Volatile-ttl: select the expired data from the dataset (server. DB [I]. Expires) that will expire.

Volatile-random: arbitrarily select data elimination from a dataset with an expiration time set (server. DB [I]. Expires).

Allkeys-lru: select the least recently used data elimination from the dataset (server. DB [I]. Dict).

Allkeys-random: data elimination from any selection of the dataset (server. DB [I]. Dict)

No-enviction (expulsion): prohibition of eviction data

When the mem_used memory has exceeded the maxmemory setting, for all read and write requests, the redis.c/freeMemoryIfNeeded (void) function is triggered to clean up the excess memory. Note that the cleanup process is blocked until enough memory space is cleared. So if the maxmemory is reached and the caller is still writing, the active cleanup policy may be triggered repeatedly, resulting in a certain delay in the request.

During cleaning, appropriate cleaning will be done according to the maxmemory-policy configured by the user (usually LRU or TTL). The LRU or TTL policy here does not aim at all key of redis, but takes the maxmemory-samples key in the configuration file as the sample pool for sample cleaning.

Summary and memo

If a large number of Key expires every day in Redis (for example, tens of millions), then you must consider cleaning up the expired Key:

Increase the frequency of Redis active cleanup (by increasing the hz parameter)

The easiest way to manually clean up expired Key is to perform scan operation. Scan operation will trigger * passive deletion. Don't forget to add count during scan operation.

The number of Key returned by the dbsize command, including expired Key

The Key returned by the randomkey command, excluding expired Key

The Key returned by the scan command, including expired Key

# Keyspace returned by the info command:

Db6:keys=1034937352,expires=994731489,avg_ttl=507838502

The number of Key corresponding to keys is equal to that of dbsize

Expires refers to the number of Key with expiration time set

Avg_ttl refers to the average expiration time of a Key with an expiration time set (in milliseconds)

At this point, I believe you have a deeper understanding of "how to clean up useless data in Redis". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.