Shulou(Shulou.com)06/01 Report--

How does redis set the expiration time? I believe that many novice rookies have not yet learned this skill. Through the summary of this article, I hope you can learn this skill. The following materials are the steps for implementation.

1. Expiration time of key in Redis

Set the expiration time of the data through the EXPIRE key seconds command. A return of 1 indicates that the setting is successful, and a return of 0 indicates that key does not exist or the expiration time cannot be set successfully. When the expiration time is set on the key, the key is automatically deleted after the specified number of seconds. Key with an expiration time specified is said to be unstable in Redis.

The expiration time associated with key will be cleared when it is deleted by the DEL command or reset by the SET or GETSET command.

127.0.0.1 ttl 6379 > setex s 20 1OK127.0.0.1:6379 > ttl s (integer) 17127.0.0.1 setex 1OK127.0.0.1:6379 > ttl s (integer) 195127.0.0.1 setex 6379 > setrange s 3100 (integer) 6127.0.0.1 ttl 6379 > ttl s (integer) 152127.0.1ttl 6379 > get s "1\ x00\ x00100" 127.0.0.1Ze6379 > ttl s (integer) 108127.0.0. 1ttl 6379 > getset s 200 "1\ x00\ x00100" 127.0.0.1 ttl 6379 > ttl s (integer)-1

Use PERSIST to clear the expiration time

127.0.0.1 get 6379 > setex s 100 testOK127.0.0.1:6379 > get s "test" 127.0.0.1 get 6379 > ttl s (integer) 94127.0.0.1 ttl 6379 > type sstring127.0.0.1:6379 > strlen s (integer) 4127.0.16379 > persist s (integer) 1127.0.0.1 integer 6379 > ttl s (integer)-1127.0.0.1 integer > get s "test"

Using rename only changed the key value.

127.0.0.1 expire 6379 > ttl s 200 (integer) 1127.0.0.1 ttl 6379 > integer) 198127.0.0.1 ttl 6379 > rename s ssOK127.0.0.1:6379 > ttl ss (integer) 187127.0.1 ttl 6379 > type ssstring127.0.0.1:6379 > get ss "test"

Note: after Redis2.6, the expire precision can be controlled within 0 to 1 millisecond, and the expiration information of key is stored in the form of absolute Unix timestamp (millisecond precision after Redis2.6), so when synchronizing multiple servers, be sure to synchronize the time of each server.

2. Redis expired key deletion policy

There are three ways for Redis key to expire:

(1) passive deletion: when reading / writing an expired key, the lazy delete policy will be triggered and the expired key will be deleted directly

(2) active deletion: since lazy deletion policy cannot guarantee that cold data will be deleted in time, Redis will take the initiative to eliminate a batch of expired key on a regular basis.

(3) active cleanup policy is triggered when the current used memory exceeds the maxmemory limit

Passive deletion

Only when the key is operated (such as GET), REDIS passively checks whether the key expires, deletes it if it expires, and returns NIL.

1. This deletion strategy is friendly to CPU. Deletion will only be carried out if necessary, and unnecessary CPU time will not be wasted on other expire key.

2. But this strategy is not memory-friendly. A key has expired, but it will not be deleted until it is operated, and it still takes up memory space. If a large number of expired keys exist but are rarely accessed, it will result in a lot of waste of memory space. The expireIfNeeded (redisDb * db, robj * key) function is located in src/db.c.

/ *-* Expires API * -- * / int removeExpire (redisDb * db Robj * key) {/ * An expire may only be removed if there is a corresponding entry in the * main dict. Otherwise, the key will never be freed. * / redisAssertWithInfo (NULL,key,dictFind (db- > dict,key- > ptr)! = NULL); return dictDelete (db- > expires,key- > ptr) = = DICT_OK;} void setExpire (redisDb * db, robj * key, long long when) {dictEntry * kde, * de; / * Reuse the sds from the main dict in the expire dict * / kde = dictFind (db- > dict,key- > ptr); redisAssertWithInfo (NULL,key,kde! = NULL) De = dictReplaceRaw (db- > expires,dictGetKey (kde)); dictSetSignedIntegerVal (de,when);} / * Return the expire time of the specified key, or-1 if no expire * is associated with this key (i.e. The key is non volatile) * / long long getExpire (redisDb * db, robj * key) {dictEntry * de; / * No expire? Return ASAP * / if (dictSize (db- > expires) = = 0 | (de = dictFind (db- > expires,key- > ptr)) = = NULL) return-1; / * The entry was found in the expire dict, this means it should also * be present in the main dict (safety check). * / redisAssertWithInfo (NULL,key,dictFind (db- > dict,key- > ptr)! = NULL); return dictGetSignedIntegerVal (de);} / * Propagate expires into slaves and the AOF file. * When a key expires in the master, a DEL operation for this key is sent * to all the slaves and the AOF file if enabled. * * This way the key expiry is centralized in one place, and since both * AOF and the master- > slave link guarantee operation ordering, everything * will be consistent even if we allow write operations against expiring * keys. * / void propagateExpire (redisDb * db, robj * key) {robj * argv [2]; argv [0] = shared.del; argv [1] = key; incrRefCount (argv [0]); incrRefCount (argv [1]); if (server.aof_state! = REDIS_AOF_OFF) feedAppendOnlyFile (server.delCommand,db- > id,argv,2); replicationFeedSlaves (server.slaves,db- > id,argv,2); decrRefCount (argv [0]) DecrRefCount (argv [1]);} int expireIfNeeded (redisDb * db, robj * key) {mstime_t when = getExpire (db,key); mstime_t now; if (when)

< 0) return 0; /* No expire for this key */ /* Don't expire anything while loading. It will be done later. */ if (server.loading) return 0; /* If we are in the context of a Lua script, we claim that time is * blocked to when the Lua script started. This way a key can expire * only the first time it is accessed and not in the middle of the * script execution, making propagation to slaves / AOF consistent. * See issue #1525 on Github for more information. */ now = server.lua_caller ? server.lua_time_start : mstime(); /* If we are running in the context of a slave, return ASAP: * the slave key expiration is controlled by the master that will * send us synthesized DEL operations for expired keys. * * Still we try to return the right information to the caller, * that is, 0 if we think the key should be still valid, 1 if * we think the key is expired at this time. */ if (server.masterhost != NULL) return now >

When; / * Return when this key has not expired * / if (now id); return dbDelete (db,key) } / *-- * Expires Commands *-- -- * / * This is the generic command implementation for EXPIRE PEXPIRE, EXPIREAT * and PEXPIREAT. Because the commad second argument may be relative or absolute * the "basetime" argument is used to signal what the basetime is (either 0 * for * AT variants of the command, or the current time for relative expires) * * unit is either UNIT_SECONDS or UNIT_MILLISECONDS, and is only used for * the argv [2] parameter. The basetime is always specified in milliseconds. * / void expireGenericCommand (redisClient * c, long long basetime, int unit) {robj * key = c-> argv [1], * param = c-> argv [2]; long long when; / * unix time in milliseconds when the key will expire. * / if (getLongLongFromObjectOrReply (c, param, & when, NULL)! = REDIS_OK) return; if (unit = = UNIT_SECONDS) when * = 1000; when + = basetime; / * No key, return zero. * / if (lookupKeyRead (c-> db,key) = = NULL) {addReply (cMagneShared.czero); return;} / * EXPIRE with negative TTL, or EXPIREAT with a timestamp into the past * should never be executed as a DEL when load the AOF or in the context * of a slave instance. * * Instead we take the other branch of the IF statement setting an expire * (possibly in the past) and wait for an explicit DEL from the master. * / if (when db,key); server.dirty++; / * Replicate/AOF this as an explicit DEL. * / aux = createStringObject ("DEL", 3); rewriteClientCommandVector (cmae2); decrRefCount (aux); signalModifiedKey (c-> db,key); notifyKeyspaceEvent (REDIS_NOTIFY_GENERIC, "del", key,c- > db- > id); addReply (cmaged.cone); return;} else {setExpire (c-> db,key,when); addReply (cmagshared.cone) SignalModifiedKey (c-> db,key); notifyKeyspaceEvent (REDIS_NOTIFY_GENERIC, "expire", key,c- > db- > id); server.dirty++; return;}} void expireCommand (redisClient * c) {expireGenericCommand (cdint mstime (), UNIT_SECONDS);} void expireatCommand (redisClient * c) {expireGenericCommand (cd0pr);} void pexpireCommand (redisClient * c) {expireGenericCommand (cMagna mstime (), UNIT_MILLISECONDS) } void pexpireatCommand (redisClient * c) {expireGenericCommand (redisClient * c, int output_ms) {long long expire, ttl =-1; / * If the key does not exist at all, return-2 * / if (lookupKeyRead (c-> db,c- > argv [1]) = = NULL) {addReplyLongLong (cMagne2); return;} / * The key exists. Return-1 if it has no expire, or the actual * TTL value otherwise. * / expire = getExpire (c-> db,c- > argv [1]); if (expire! =-1) {ttl = expire-mstime (); if (ttl)

< 0) ttl = 0; } if (ttl == -1) { addReplyLongLong(c,-1); } else { addReplyLongLong(c,output_ms ? ttl : ((ttl+500)/1000)); }} void ttlCommand(redisClient *c) { ttlGenericCommand(c, 0);} void pttlCommand(redisClient *c) { ttlGenericCommand(c, 1);} void persistCommand(redisClient *c) { dictEntry *de; de = dictFind(c->

Db- > dict,c- > argv [1]-> ptr); if (de = = NULL) {addReply (CMH shared.czero);} else {if (removeExpire (c-> db,c- > argv [1])) {addReply (CMH shared.cone); server.dirty++;} else {addReply (CMH shared.czero);}

But this alone is not enough, because there may be some key that will never be accessed again, and these key with expiration time also need to be deleted after expiration. We can even think of this as a memory leak-useless junk data takes up a lot of memory, but the server will not release it on its own, which is very dependent on memory for Redis servers. It's definitely not good news.

Active deletion

Let's talk about time events first. for continuously running servers, the server needs to check and organize its own resources and status regularly so as to keep the server in a healthy and stable state, which is collectively referred to as cron job.

In Redis, general operations are implemented by redis.c/serverCron, which mainly performs the following operations:

Update all kinds of statistics of the server, such as time, memory footprint, database footprint, etc.

Clean up expired key-value pairs in the database.

Resize unreasonable databases.

Close and clean up clients with failed connections.

Try an AOF or RDB persistence operation.

If the server is the primary node, periodically synchronize the subsidiary nodes.

If you are in cluster mode, perform regular synchronization and connectivity tests on the cluster.

Redis runs serverCron as a time event, ensuring that it runs automatically at regular intervals, and because serverCron needs to run periodically while the Redis server is running, it is a cyclical time event: serverCron executes periodically until the server shuts down.

In Redis 2.6, the program requires serverCron to run 10 times per second, an average of once every 100ms. Starting with Redis 2. 8, users can adjust the number of serverCron execution per second by modifying the hz option.

Also known as scheduled deletion, the term "periodic" here refers to the cleaning policy that is periodically triggered by Redis, which is done by the activeExpireCycle (void) function located in src/redis.c.

ServerCron is a positioning task driven by redis's event framework. In this scheduled task, the activeExpireCycle function is called to delete expired key for each db within the limited time REDIS_EXPIRELOOKUPS_TIME_LIMIT. The reason for limiting the time limit is to prevent long-term blocking from affecting the normal operation of the redis. This active deletion strategy makes up for the memory unfriendliness of the passive deletion strategy.

Therefore, Redis will periodically randomly test a batch of key with expiration time set and process them. Expired key tested will be deleted. Typically, Redis does the following steps 10 times per second:

(1) randomly test 100 key with expiration time set

(2) Delete all expired key found

(3) if more than 25 key are deleted, repeat step 1.

This is a simple algorithm based on probability. The basic assumption is that the extracted samples can represent the entire key space, and redis continues to clean up expired data until the percentage of key that is about to expire is reduced to less than 25%. This also means that the maximum amount of key that has expired but still occupies memory space at any given time is the number of writes per second divided by 4. 5%.

The default value in Redis-3.0.0 is 10, which means that background tasks are called 10 times per second.

In addition to the frequency of active elimination, Redis also has a limit on the maximum duration of each elimination task, which ensures that there will not be too many blocking application requests for each active elimination. Here is the limited calculation formula:

# define ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC 25 / * CPU max% for keys collection * /. Timelimit = 1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100

Scaling up hz will increase the frequency of active elimination of Redis. If your Redis storage contains a lot of cold data that takes up too much memory, you can consider increasing this value, but the Redis authors recommend that this value not exceed 100. We actually increased this value to 100 online and observed that CPU increased by about 2%, but there was a significant increase in memory release speed for cold data (by observing the number of keyspace and used_memory size).

You can see that the relationship between timelimit and server.hz is reciprocal, that is, the larger the hz configuration, the smaller the timelimit. In other words, the higher the expected frequency of active elimination per second, the shorter the maximum duration of each phase-out. Here, the maximum phase-out time per second is a fixed 250ms (1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/100), while the phase-out frequency and the maximum time of each phase-out are controlled by the hz parameter.

From the above analysis, when the expired key ratio in the redis does not exceed 25%, increasing the hz can significantly increase the minimum number of scanned key. Assuming that hz is 10, at least 200 key are scanned in a second (10 calls per second * at least 20 key are randomly selected each time). If hz is changed to 100, then at least 2000 key; are scanned in one second. On the other hand, if the expired key ratio exceeds 25%, there is no limit to the number of scanned key, but the cpu time takes up 250ms at most per second.

When REDIS is running in master-slave mode, only the master node executes these two expired deletion policies, and then synchronizes the deletion operation "del key" to the slave node.

Maxmemory

When the current used memory exceeds the maxmemory limit, trigger the active cleanup policy:

Volatile-lru: LRU only key with expiration time set (default)

Allkeys-lru: delete the key of the lru algorithm

Volatile-random: randomly delete an expiring key

Allkeys-random: random deletion

Volatile-ttl: delete the expiring

Noeviction: never expires, return error when mem_used memory has exceeded the maxmemory setting, for all read and write requests, the redis.c/freeMemoryIfNeeded (void) function will be triggered to clean up the excess memory. Note that the cleanup process is blocked until enough memory space is cleared. So if the maxmemory is reached and the caller is still writing, the active cleanup policy may be triggered repeatedly, resulting in a certain delay in the request.

When the mem_used memory has exceeded the maxmemory setting, for all read and write requests, the redis.c/freeMemoryIfNeeded (void) function is triggered to clean up the excess memory. Note that the cleanup process is blocked until enough memory space is cleared. So if the maxmemory is reached and the caller is still writing, the active cleanup policy may be triggered repeatedly, resulting in a certain delay in the request.

During cleaning, appropriate cleaning will be done according to the maxmemory-policy configured by the user (usually LRU or TTL). The LRU or TTL policy here does not aim at all key of redis, but takes the maxmemory-samples key in the configuration file as the sample pool for sample cleaning.

The default configuration of maxmemory-samples in redis-3.0.0 is 5. If you increase it, it will improve the accuracy of LRU or TTL. The result of the redis author's test is that when this configuration is 10:00, it is very close to the accuracy of full LRU, and increasing maxmemory-samples will lead to more CPU time consumed during active cleaning. It is recommended:

(1) do not trigger maxmemory as far as possible. After the memory consumption of mem_used reaches a certain proportion of maxmemory, you need to consider increasing hz to speed up phase-out, or expand cluster capacity.

(2) if you can control the memory, you do not need to modify the maxmemory-samples configuration; if Redis itself is a LRU cache service (this service is usually in the maxmemory state for a long time, and Redis automatically does LRU elimination), you can scale up the maxmemory-samples appropriately.

The following is a description of the configuration parameters mentioned above

# Redis calls an internal function to perform many background tasks, like # closing connections of clients in timeout, purging expired keys that are # never requested, and so forth. # # Not all tasks are performed with the same frequency, but Redis checks for # tasks to perform according to the specified "hz" value. # # By default "hz" is set to 10.Raising the value will use more CPU when # Redis is idle, but at the same time will make Redis more responsive when # there are many keys expiring at the same time, and timeouts may be # handled with more precision. # # The range is between 1 and 500, however a value over 100 is usually not # a good idea. Most users should use the default of 10 and raise this up to # 100 only in environments where very low latency is required. Hz 10 # MAXMEMORY POLICY: how Redis will select what to remove when maxmemory # is reached. You can select among five behaviors: # # volatile-lru-> remove the key with an expire set using an LRU algorithm # allkeys-lru-> remove any key according to the LRU algorithm # volatile-random-> remove a random key with an expire set # allkeys-random-> remove a random key, any key # volatile-ttl-> remove the key with the nearest expire time (minor TTL) # noeviction-> don't expire at all, just return an error on write operations # # Note: with any of the above policies, Redis will return an error on write # operations When there are no suitable keys for eviction. # # At the date of writing these commands are: set setnx setex append # incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd # sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby # zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby # getset mset msetnx exec sort # # The default is: # maxmemory-policy noeviction # LRU and minimal TTL algorithms are not precise algorithms but approximated # algorithms (in order to save memory), so you can tune it for speed or # accuracy For default Redis will check five keys and pick the one that was # used less recently, you can change the sample size using the following # configuration directive. # # The default of 5 produces good enough results. 10 Approximates very closely # true LRU but costs a bit more CPU. 3 is very fast but not very accurate. # maxmemory-samples 5

Expiration handling in Replication link and AOF files

In order to get the correct behavior without causing consistency problems, when a key expires, the DEL operation is recorded in the AOF file and passed to all relevant slave. In other words, the expired deletion operation is uniformly carried out in the master instance and passed down, rather than controlled by each salve.

In this way, there will be no data inconsistencies. When slave connects to master, it cannot immediately clean up the expired key (you need to wait for the DEL operation passed by master). Slave still needs to manage and maintain the expired state in the dataset so that when slave is promoted to master, it can handle expiration independently like master.

After reading this article, have you learned how to set the expiration time by redis? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel. Thank you for reading.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.