Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "What is Redis's expiration strategy". In the operation process of actual cases, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

Save expiration time

Redis can set an expiration time for each key and places each key with an expiration time in a separate dictionary.

typedef struct redisDb { int id; //id is the database serial number, 0-15 (default Redis has 16 databases) long avg_ttl; //average ttl of database objects stored (time to live), used to count dict *dict; //store all key-values dict *expires in the database; //store the expiration time of keys dict *blocking_keys;//bpop store blocking keys and client objects dict *ready_keys;//after blocking push response blocking client stores key of push after blocking and client object dict *watched_keys;//store key of watch monitored and client object} redisDb;

dict is used to maintain all Key-Value pairs contained in a Redis database, and expires is used to maintain a key with an expiration time set in the Redis database (i.e., the mapping of key and expiration time). Note that the expiration time here is represented by a timestamp of milliseconds, for example, if 2022-01-02 22:45:02 expires, the value is 1641134702000.

When we use expire to set the expiration time of a key, Redis first looks up whether the key to be set exists in the dictionary table, and if it exists, adds the key and expiration time to the dictionary table.

When we insert data into the system using the setex command, Redis first adds Key and Value to the dictionary table dict, and then adds Key and expiration time to the dictionary table expires. Note that setex can only be used with strings.

To sum up simply, the key and specific failure time that set the failure time are all maintained in the expires dictionary table.

Set expiration time

Use of Expire

The expire command can be used as follows: expire key ttl(in seconds)

127.0.0.1: 6379> expire name 2 #2 seconds expire (integer) 1 127.0.0.1: 6379> get name (nil) 127.0.0.1: 6379> set name zhangfei OK 127.0.0.1: 6379> ttl name #permanently valid (integer) -1 127.0.0.1: 6379> expire name 30 #30 seconds expire (integer) 1 127.0.0.1: 6379> ttl name #24 seconds expire (integer) 24 127.0.0.1: 6379> ttl name #expire (integer) -2

Redis has four different commands that you can use to set a key's lifetime (how long the key will live) or expiration time (when the key will be deleted):

The expire command is used to set the key key's lifetime to ttl seconds

The pexpire command sets the key key's lifetime to ttl milliseconds

The expire command sets the expiration time of the key to the number of seconds specified by timestamp

The pexpire command sets the key expiration time to the number of milliseconds specified by timestamp

Note that the final implementation of expire, pexpire and expire is implemented through pexpire, that is to say, no matter which command the client executes, Redis will be converted to pexpire command execution. So the time stored in the expires dictionary is the expiration time of the key represented by the millisecond timestamp.

expiration policy

If a key expires, when is it deleted?

There are three expiration strategies

Timed deletion: While setting the key expiration time, create a timer that will immediately delete the key when the key expiration time comes. (Create Timer Delete)

Lazy Delete: Allow the key to expire, but each time a key is retrieved from the keyspace, check whether the retrieved key is expired, and if it is expired, delete the key; if it is not expired, return the key. (deleted when used)

Periodically delete: Every once in a while, the program checks the database to remove expired keys. Algorithms determine how many expired keys to delete and how many databases to check. (Scan regularly to delete)

timed deletion

advantages

1, the most memory-friendly: through the use of timers, you can ensure that expired keys will be deleted as soon as possible, releasing the occupied memory.

disadvantages

1. Most unfriendly to CPU: In the case of many expired keys, deleting the expired key may take up a considerable part of CPU time and affect the server response time and throughput.

lazy deletion

advantages

1, the most friendly to the CPU: only when the key is removed will the expired key be checked, that is, there is no need for the CPU to scan regularly, and there is no need to create a large number of timers.

disadvantages

Most memory-unfriendly: If a key has expired but will not be accessed later, it will remain in the database. If there are too many such keys, it will undoubtedly take up a lot of memory.

periodically delete

Periodic deletion is a compromise between timed deletion and lazy deletion above.

advantages

Periodically delete the expired key operation every once in a while, and reduce the impact of deletion operations on CPU time by limiting the length and frequency of deletion operations.

2. By deleting the expired key, the memory waste caused by the expired key can be effectively reduced.

Disadvantages Difficult to determine how long and how often deletion operations are performed

If delete operations are performed too frequently or for too long, the periodic delete policy degenerates into timed deletes that take up too much CPU execution time.

2. If the deletion operation takes too little time to execute, or the execution time is too short, the periodic deletion policy will be the same as lazy deletion, resulting in memory waste.

Redis expiration policy

Redis uses two strategies: lazy deletion and periodic deletion: by matching these two strategies, the server can strike a good balance between using cpu time wisely and avoiding wasting memory space.

Implementation of Lazy Deletion Policy

The policy of lazy deletion of expired keys is implemented by db.c/expireIfNeeded function. All Redis commands that read and write database will call expireIfNeed function to check the input key before execution:

If the key has expired, the expireIfNeeded function deletes the key

If the key has not expired, the expireIfNeeded function does nothing.

The process of invoking expire IfNeeded function by command is as follows

In addition, because every key accessed can be deleted, each command must be able to handle both the presence and absence of keys. The following diagram shows the execution process of the get command

Implementation of Periodic Deletion Policy

The periodic deletion policy of expired keys is implemented by the redis.c/activeExpireCycle function, which is called whenever the server periodically operates redis.c/serverCron function is executed, and it traverses each database in the server several times within a specified time.

Redis defaults to 10 expiration scans per second. Expiration scans do not traverse all keys in the expiration dictionary. Instead, they use a simple greedy policy, as follows.

(1)Choose 20 keys randomly from the expired dictionary.

(2)Remove the expired keys from these 20 keys.

(3)If the percentage of expired keys exceeds 1/4, repeat step (1). At the same time, in order to ensure that the expired scan will not appear excessive cycle, resulting in the phenomenon of final stuck, the algorithm also increased the upper limit of the scan time, the default will not exceed 25 ms.

Suppose all keys in a large Redis instance expire at the same time, what happens?

consuming CPU

Redis continues to scan the outdated dictionary (loop multiple times) until the outdated keys in the outdated dictionary become sparse (loop times decrease significantly).

Causes requests to stall or timeout

When the client request arrives, if the server enters the expired scan state, the client request will wait at least 25ms before processing. If the client sets the timeout time to be relatively short, such as 10ms, a large number of connections will be closed due to timeout, and many exceptions will occur at the service end.

So be sure to pay attention to the expiration time. If a large number of keys expire, set a random range for the expiration time, and not all expire at the same time.

"What is Redis 'expiration strategy" is introduced here. Thank you for reading it. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.