Shulou(Shulou.com)05/31 Report--

This article mainly introduces the example analysis of expiration operation and expiration policy in Redis, which is very detailed and has certain reference value. Friends who are interested must read it all!

Expired operation

Expiration settin

The expiration time in Redis is mainly set in the following four ways:

Expire key seconds: sets key to expire in n seconds.

Pexpire key milliseconds: sets key to expire in n milliseconds.

Expireat key timestamp: sets the key to expire after a certain timestamp (accurate to seconds).

Pexpireat key millisecondsTimestamp: sets the key to expire after a certain timestamp (accurate to milliseconds).

You can use the command ttl key (in seconds) or pttl key (in milliseconds) to see how long key is out of date.

Redis can use the time command to query the timestamp of the current time (accurate to seconds).

There are several methods for directly manipulating the expiration time in a string, as listed below:

Set key value ex seconds: set the key-value pair and specify the expiration time (accurate to seconds).

Set key value px milliseconds: sets the key-value pair while specifying the expiration time (accurate to milliseconds).

Setex key seconds valule: set the key-value pair and specify the expiration time (accurate to seconds).

Remove expiration time

Use the command: persist key can remove the expiration time of the key value. -1 means it will never expire.

Java implements expiration operation

Use Jedis to implement the operation on Redis, code:

Public class TTLTest {public static void main (String [] args) throws InterruptedException {/ / create a Redis connection Jedis jedis = new Jedis ("xxx.xxx.xxx.xxx", 6379); / / set the Redis password (if there is no password, this line can be omitted) jedis.auth ("xxx") / / store key-value pairs (never expire by default) jedis.set ("k", "v"); / query TTL (expiration time) Long ttl = jedis.ttl ("k"); / / print expiration log / / expiration time:-1 System.out.println ("expiration time:" + ttl) / / set the expired jedis.expire after 100s ("k", 100); / / wait 1 s and then execute Thread.sleep (1000); / / print the expired log / / execute the TTL=99 System.out.println after expire ("TTL= after expire" + jedis.ttl ("k"));}}

For more expiration methods, please see the following list:

Pexpire (String key, long milliseconds): expires after setting n milliseconds.

ExpireAt (String key, long unixTime): expires after setting a timestamp (accurate to seconds).

PexpireAt (String key, long millisecondsTimestamp): expires after setting a timestamp (accurate to milliseconds).

Persist (String key): removes expiration time.

Public class TTLTest {public static void main (String [] args) throws InterruptedException {/ / create a Redis connection Jedis jedis = new Jedis ("xxx.xxx.xxx.xxx", 6379); / / set the Redis password (if there is no password, this line can be omitted) jedis.auth ("xxx") / / store key-value pairs (never expire by default) jedis.set ("k", "v"); / / query TTL (expiration time) Long ttl = jedis.ttl ("k"); / / print expiration log System.out.println ("expiration time:" + ttl) / / set the expired jedis.expire after 100s ("k", 100); / / wait 1 s and then execute Thread.sleep (1000); / / print the expired log System.out.println ("TTL= after executing expire" + jedis.ttl ("k")); / / set the expired jedis.pexpire after n milliseconds ("k", 100000) / / set a timestamp to expire (accurate to seconds) jedis.expireAt ("k", 1573468990); / / set a timestamp to expire (accurate to milliseconds) jedis.pexpireAt ("k", 15734689900L); / / remove the expiration time jedis.persist ("k");}}

Expired keys in persistence

Expired keys in RDB

The RDB file is divided into two phases, the RDB file generation phase and the loading phase.

1. RDB file generation

RDB loading is divided into the following two situations:

If Redis is the main server running mode, when loading the RDB file, the program will check the keys saved in the file, and the expired keys will not be loaded into the database. So expired keys will not affect the primary server that loads the RDB file.

If Redis runs from the server, it will be loaded into the database when the RDB file is loaded, regardless of whether the key has expired or not. However, because the master-slave server is synchronizing, the data of the slave server will be emptied. So in general, expired keys have no effect on the slave server that loads the RDB file.

The source code loaded in the RDB file can be found in the rdbLoad () function of the rdb.c file, as shown in the source code:

/ * Check if the key already expired. This function is used when loading* an RDB file from disk, either at startup, or when an RDB was* received from the master. In the latter case, the master is* responsible for key expiry. If we would expire keys here, the* snapshot taken by the master may not be reflected on the slave. * * if the server is the primary node, * then when the keys have expired, they are no longer associated with the database * / if (server.masterhost = = NULL & & expiretime! =-1 & & expiretime

< now) { decrRefCount(key); decrRefCount(val); // 跳过 continue;}AOF 中的过期键 1. AOF 文件写入 当 Redis 以 AOF 模式持久化时，如果数据库某个过期键还没被删除，那么 AOF 文件会保留此过期键，当此过期键被删除后，Redis 会向 AOF 文件追加一条 DEL 命令来显式地删除该键值。 2. AOF 重写 执行 AOF 重写时，会对 Redis 中的键值对进行检查已过期的键不会被保存到重写后的 AOF 文件中，因此不会对 AOF 重写造成任何影响。 主从库的过期键 当 Redis 运行在主从模式下时，从库不会进行过期扫描，从库对过期的处理是被动的。也就是即使从库中的 key 过期了，如果有客户端访问从库时，依然可以得到 key 对应的值，像未过期的键值对一样返回。 从库的过期键处理依靠主服务器控制，主库在 key 到期时，会在 AOF 文件里增加一条 del 指令，同步到所有的从库，从库通过执行这条 del 指令来删除过期的 key。 过期策略 在 Redis 中我们可以给一些元素设置过期时间，那当它过期之后 Redis 是如何处理这些过期键呢？ 过期键执行流程 Redis 之所以能知道那些键值过期，是因为在 Redis 中维护了一个字典，存储了所有设置了过期时间的键值，我们称之为过期字典。 过期键源码分析 过期键存储在 redisDb 结构中，源代码在 src/server.h 文件中（基于 Redis 5）： /* Redis database representation. There are multiple databases identified * by integers from 0 (the default database) up to the max configured * database. The database number is the 'id' field in the structure. */typedef struct redisDb { dict *dict; /* 数据库键空间，存放着所有的键值对 */ dict *expires; /* 键的过期时间 */ dict *blocking_keys; /* Keys with clients waiting for data (BLPOP)*/ dict *ready_keys; /* Blocked keys that received a PUSH */ dict *watched_keys; /* WATCHED keys for MULTI/EXEC CAS */ int id; /* Database ID */ long long avg_ttl; /* Average TTL, just for stats */ list *defrag_later; /* List of key names to attempt to defrag one by one, gradually. */} redisDb; 过期键数据结构如下图所示： 过期策略 Redis 会删除已过期的键值，以此来减少 Redis 的空间占用，但因为 Redis 本身是单线的，如果因为删除操作而影响主业务的执行就得不偿失了，为此 Redis 需要制定多个（过期）删除策略来保证正常执行的性能。 定时删除 在设置键值过期时间时，创建一个定时事件，当过期时间到达时，由事件处理器自动执行键的删除操作。 优点：保证内存可以被尽快地释放。 缺点：在 Redis 高负载的情况下或有大量过期键需要同时处理时，会造成 Redis 服务器卡顿，影响主业务执行。 惰性删除 不主动删除过期键，每次从数据库获取键值时判断是否过期，如果过期则删除键值，并返回 null。 优点：因为每次访问时，才会判断过期键，所以此策略只会使用很少的系统资源。 缺点：系统占用空间删除不及时，导致空间利用率降低，造成了一定的空间浪费。 源码解析 惰性删除的源码位于 src/db.c 文件的 expireIfNeeded 方法中，源码如下： int expireIfNeeded(redisDb *db, robj *key) { // 判断键是否过期 if (!keyIsExpired(db,key)) return 0; if (server.masterhost != NULL) return 1; /* 删除过期键 */ // 增加过期键个数 server.stat_expiredkeys++; // 传播键过期的消息 propagateExpire(db,key,server.lazyfree_lazy_expire); notifyKeyspaceEvent(NOTIFY_EXPIRED, "expired",key,db->

Id); / / a server.lazyfree_lazy_expire of 1 means asynchronous deletion (lazy space release), otherwise delete return server.lazyfree_lazy_expire synchronously? DbAsyncDelete (db,key): dbSyncDelete (db,key);} / / determine whether the key expires int keyIsExpired (redisDb * db, robj * key) {mstime_t when = getExpire (db,key); if (when)

< 0) return 0; /* No expire for this key */ /* Don't expire anything while loading. It will be done later. */ if (server.loading) return 0; mstime_t now = server.lua_caller ? server.lua_time_start : mstime(); return now >

When;} / / get the expiration time of the key long long getExpire (redisDb * db, robj * key) {dictEntry * de; / * No expire? Return ASAP * / if (dictSize (db- > expires) = = 0 | (de = dictFind (db- > expires,key- > ptr)) = = NULL) return-1; / * The entry was found in the expire dict, this means it should also * be present in the main dict (safety check). * / serverAssertWithInfo (NULL,key,dictFind (db- > dict,key- > ptr)! = NULL); return dictGetSignedIntegerVal (de);}

Before all read and write commands to the database are executed, the expireIfNeeded method is called to determine whether the key value has expired, and if it expires, it will be deleted from the database, otherwise no processing will be done.

Delete periodically

Check the database at regular intervals and randomly delete some expired keys.

Redis defaults to 10 expiration scans per second. This configuration can be configured through Redis's configuration file redis.conf, and the configuration key is hz. Its default value is hz 10.

Note: instead of traversing all the keys in the expired dictionary, each Redis scan is performed in the form of random extraction to determine and delete expired keys.

Delete the process periodically

Take 20 keys randomly from the expired dictionary.

Delete expired keys from these 20 keys.

If the percentage of expired key exceeds 25%, repeat step 1.

At the same time, in order to ensure that the expired scan will not cycle too much, resulting in thread jam, the algorithm also increases the upper limit of scanning time, which will not exceed 25ms by default.

Advantages: by limiting the duration and frequency of deletion operations, to reduce the impact of deletion operations on the main business of Redis, but also delete part of the expired data to reduce the invalid occupation of space by expired keys.

Disadvantages: memory cleanup is not as effective as regular deletion, and less system resources are used without lazy deletion.

Source code parsing

The core source code that is deleted periodically is found in the activeExpireCycle method under the src/expire.c file. The source code is as follows:

Void activeExpireCycle (int type) {static unsigned int current_db = 0; / * the database traversed to the last periodic deletion ID * / static int timelimit_exit = 0; / * Time limit hit in previous call? * / static long long last_fast_cycle = 0; / * the last point in time to perform a fast periodic deletion * / int j, iteration = 0; int dbs_per_call = CRON_DBS_PER_CALL / / the number of databases that are periodically deleted each time, long long start = ustime (), timelimit, elapsed; if (clientsArePaused ()) return; if (type = = ACTIVE_EXPIRE_CYCLE_FAST) {if (! timelimit_exit) return; / / ACTIVE_EXPIRE_CYCLE_FAST_DURATION is the execution time if of fast periodic deletion (start)

< last_fast_cycle + ACTIVE_EXPIRE_CYCLE_FAST_DURATION*2) return; last_fast_cycle = start; } if (dbs_per_call >

Server.dbnum | | timelimit_exit) dbs_per_call = server.dbnum; / / duration of execution of slow periodic deletions timelimit = 1000000 active active expired expired CYCLEX sloping timed PERC num 100; timelimit_exit = 0; if (timelimit ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP) time = ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP / / randomly select num keys from the database to check while (num--) {dictEntry * de; long long ttl; if ((de = dictGetRandomKey (db- > expires)) = = NULL) break Ttl = dictGetSignedInteger / / Expiration check, and delete the expired key if (activeExpireCycleTryExpire (db,de,now)) expired++; if (ttl > 0) {/ * We want the average TTL of keys yet not expired. * / ttl_sum + = ttl; ttl_samples++;} total_sampled++;} total_expired + = expired; if (ttl_samples) {long long avg_ttl = ttl_sum/ttl_samples If (db- > avg_ttl = = 0) db- > avg_ttl = avg_ttl; db- > avg_ttl = (db- > avg_ttl/50) * 49 + (avg_ttl/50);} if ((iteration & 0xf) = = 0) {/ * check once every 16 iterations. * / elapsed = ustime ()-start; if (elapsed > timelimit) {timelimit_exit = 1; server.stat_expired_time_cap_reached_count++; break }} / * only ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4 expired keys are deleted per check * /} while (expired > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4);} / /.}

The activeExpireCycle method traverses each database several times at a specified time, randomly checks the expiration time of some expired keys from the expired dictionary, and deletes the expired keys.

This function has two execution modes, one is fast mode and the other is slow mode, which reflects the timelimit variable in the code, which is used to restrict the running time of this function. The value of timelimit is fixed in fast mode, which is equal to the predefined constant ACTIVE_EXPIRE_CYCLE_FAST_DURATION, while in slow mode, the value of this variable is calculated by 1000000 * ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100.

Expiration policy used by Redis

Redis uses an expiration policy of lazy deletions plus periodic deletions.

The above is all the contents of the article "sample Analysis of expiration actions and expiration policies in Redis". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.