Shulou(Shulou.com)05/31 Report--

This article shares with you the content of an example analysis of the principle of redis_memcached failure. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

one. How can I trigger the failure of key?

In addition to calling the PERSIST command, is there any other situation that will undo the expiration time of a primary key? The answer is yes.

1. When you delete a primary key through the DEL command

two。 When a primary key with an expiration time set is overwritten by an update, the expiration time of the primary key is also revoked.

3. The special command is RENAME, when we use RENAME to rename a primary key, the previous associated expiration time will be automatically passed to the new primary key, but if a primary key is covered by RENAME (such as the primary key hello may be overwritten by the command RENAME world hello), then the overwritten primary key expiration time will be automatically revoked, while the new primary key will continue to maintain the characteristics of the original primary key.

Note that what we are talking about here is that the primary key is overwritten by the update, not the Value corresponding to the primary key is overwritten by the update, so SET, MSET or GETSET may cause the primary key to be overwritten by the update, while INCR, DECR, LPUSH, HSET, etc., all update the corresponding values of the primary key, which will not touch the expiration time of the primary key.

two. How Redis manages and maintains primary keys

1) .Redis storage structure

Typedef struct redisDb {dict * dict;// storage primary key and value mapping dict * expires;// storage primary key and expiration time mapping dict * blocking_keys; dict * ready_keys; dict * watched_keys; int id;} redisDb

Just look at the first two structures.

Dict: used to maintain all Key-Value pairs contained in a Redis database (its structure can be understood as dict [key]: value, that is, the mapping between primary keys and values)

Expires: used to maintain a primary key with expiration time set in a Redis database (its structure can be understood as expires [key]: timeout, that is, the mapping of primary key to expiration time).

The primary key with the failure time set and the specific failure time are all maintained in the dictionary table of expires.

When we use the EXPIRE, EXPIREAT, PEXPIRE, and PEXPIREAT commands to set the expiration time of a primary key

Redis first goes to the dict dictionary table to see if the primary key you want to set exists, and if so, adds the primary key and expiration time to the expires dictionary table.

2)。 Negative method

1.expireIfNeeded function

Trigger: this function is called in any function that accesses data, and Redis calls it when implementing all commands that involve reading data, such as GET, MGET, HGET, LRANGE, etc.

Meaning: check to see if the key is invalid before reading the data, and delete it if it fails.

The 2.propagateExpire function (called in the above function) the main function

Trigger: when the last function was executed, it was in it

Meaning: used to broadcast information that the primary key has expired before it is officially deleted

Action:

(1)。 One is to send it to the AOF file to record the deletion of the invalid primary key in DEL Key's standard command format.

(2)。 All Slave sent to the current Redis server will also tell these Slave to delete their respective failed primary keys in DEL Key's standard command format.

We've seen how Redis removes failed primary keys in a negative way, but it's obviously not enough, because if some failed primary keys don't wait to be accessed again, Redis will never know that these primary keys have expired and will never delete them, which will undoubtedly lead to a waste of memory space. So there is the following way.

3)。 Positive approach: (this method is implemented by using the time event of Redis, that is, interrupting to complete some specified operations at regular intervals)

1.serverCron function:

Trigger: it is created when the Redis server starts

Function: this callback function not only checks and deletes failed primary keys, but also updates statistics, controls client connection timeouts, triggers BGSAVE and AOF, and so on.

2.activeExpireCycle function:

Trigger: when the last function is executed, the number of times per second in it is specified by the macro definition [REDIS_DEFAULT_HZ]. By default, it executes 10 times per second.

Action:

a)。 Traversing the expires dictionary table of each database in the Redis server, trying to randomly sample [REDIS_EXPIRELOOKUPS_PER_CRON] (default value is 10) primary keys with expiration time set

b)。 Check if they have failed and delete the failed primary key

c)。 If the number of failed primary keys accounts for more than 25% of the number of primary keys sampled this time, Magi Redis will think that there are still many failed primary keys in the current database, so it will continue with the next round of random sampling and deletion.

d)。 Do not stop processing the current database until the proportion is less than 25%, and move on to the next database.

Other: the activeExpireCycle function avoids invalid primary key deletion taking up too much CPU resources, so it does not attempt to process all databases in Redis at once, but only deals with a maximum of REDIS_DBCRON_DBS_PER_CALL (default is 16) libraries, with processing time limit

three. What are the similarities and differences between Memcached's method of deleting a failed primary key and Redis?

First of all, Memcached also uses a negative approach when deleting a failed primary key, that is, it does not monitor the failure of the primary key internally in Memcached, but checks to see if the primary key has expired only when it is accessed through Get.

Second, the biggest difference in the primary key failure mechanism between Memcached and Redis is that Memcached does not really delete the failed primary key like Redis, but simply reclaims the space occupied by the failed primary key. In this way, when new data is written to the system, Memcached will give priority to the space where the primary key fails. If the failed primary key runs out of space, Memcached can also use the LRU mechanism to reclaim the space that has not been accessed for a long time.

Therefore, Memcached does not require periodic deletions like those in Redis, which is also determined by the memory management mechanism used by Memcached. At the same time, it should be pointed out that when OOM appears, Redis can also configure the parameter maxmemory-policy to decide whether to use the LRU mechanism to reclaim memory space.

four. Summary:

Redis performs 10 expiration checks per second. Each time, 10 key are randomly selected from the expire table of a library to check whether it is invalid, and delete it if it fails. When the failure ratio exceeds 1 stop 4, the random sampling 10key is re-executed, not counting 1 out of 10 times, until all 10 times in this second are completed.

Q:

So someone asked, in case, in case! Failed key to do enough, 1 second of these 10 times did not finish the execution of the next second, how to do it?

A:

Redis has a detection mechanism and won't let it drag CPU to death:

a. The limit on the number of databases per processing,

The limit on the number of times the b.activeExpireCycle function executes in one second,

c. The time limit assigned to the activeExpireCycle function CPU,

d. By continuing to remove the limitation of the percentage of primary key failure, Redis has greatly reduced the impact of primary key failure mechanism on the overall performance of the system.

Therefore, we can also get the principle of setting the failure time: to avoid the failure of a large number of key at the same point in time as much as possible, it needs processing time.

The main function of negative failure. AcquiateExpire function:

Void propagateExpire (redisDb * db, robj * key) {robj * argv [2]; / / shared.del is a commonly used Redis object that has been initialized at the beginning of Redis server startup, that is, the DEL command argv [0] = shared.del; argv [1] = key; incrRefCount (argv [0]); incrRefCount (argv [1]) / / check whether AOF is enabled on the Redis server. If so, record a DEL log if (server.aof_state! = REDIS_AOF_OFF) feedAppendOnlyFile (server.delCommand,db- > id,argv,2) for the failed primary key. / / check whether the Redis server has Slave, and if so, send the command of DEL failed primary key to all Slave. This is why there is no need to actively delete the invalid primary key when it is found to be Slave in the above expireIfNeeded function, because it / / only needs to listen to the command sent by Master to OK if (listLength (server.slaves)) replicationFeedSlaves (server.slaves,db- > id,argv,2); decrRefCount (argv [0]) DecrRefCount (argv [1]);}

Active failure main function .activeExpireCycle function:

Void activeExpireCycle (void) {/ / because each call to the activeExpireCycle function does not check all Redis databases at once, you need to record the number of the last Redis database processed by each function call, so that the next call to the activeExpireCycle function / / can continue processing from this database, which is why current_db is declared as static And the other variable timelimit_exit is to record whether the execution time of the last call to activeExpireCycle function has reached / / to the time limit, so it needs to be declared as static static unsigned int current_db = 0. Static int timelimit_exit = 0; unsigned int j, iteration = 0; / / the number of Redis databases processed by each call to the activeExpireCycle function is REDIS_DBCRON_DBS_PER_CALL unsigned int dbs_per_call = REDIS_DBCRON_DBS_PER_CALL; long long start = ustime (), timelimit / / if the number of databases in the current Redis server is less than REDIS_DBCRON_DBS_PER_CALL, all databases will be processed. / / if the execution time of the last call to activeExpireCycle function reaches the time limit, there are more failed primary keys, and / / all databases if (dbs_per_call > server.dbnum | | timelimit_exit) dbs_per_call = server.dbnum will be selected. / / the longest time (in microseconds) for the execution of the activeExpireCycle function, where REDIS_EXPIRELOOKUPS_TIME_PERC / / is the proportion of CPU time that can be allocated to the execution of the activeExpireCycle function per unit time. The default value is 25centserver.hz / /, that is, the number of activeExpireCycle calls in a second, so this formula should be written more clearly like this. That is (1000000 * (REDIS_EXPIRELOOKUPS_TIME_PERC/ 100)) / server.hz timelimit = 1000000*REDIS_EXPIRELOOKUPS_TIME_PERC/server.hz/100 Timelimit_exit = 0; if (timelimit expires) = = 0) break; slots = dictSlots (db- > expires); now = mstime (); / / if the expires dictionary table is not empty but its fill rate is less than 1%, the cost of randomly selecting the primary key for checking will be very high, so check the next database if (num & slots > DICT_HT_INITIAL_SIZE & & (num*100/slots) directly here.

REDIS_EXPIRELOOKUPS_PER_CRON) num = REDIS_EXPIRELOOKUPS_PER_CRON; while (num--) {dictEntry * de; long long t; / / randomly get a primary key with a failure time set and check whether it has expired if ((de = dictGetRandomKey (db- > expires)) = = NULL) break; t = dictGetSignedIntegerVal (de) If (now > t) {/ / found that the primary key has indeed expired, delete the primary key sds key = dictGetKey (de); robj * keyobj = createStringObject (key,sdslen (key)); / / also broadcast the failure information of the primary key propagateExpire (db,keyobj); dbDelete (db,keyobj); decrRefCount (keyobj) before deletion Expired++; server.stat_expiredkeys++;}} / / add one to the iteration after each sample deletion, and check whether the execution time has reached the time limit after every 16 sample deletions. If the time limit has been reached, record the time limit reached and exit the iteration++. If ((iteration & 0xf) = = 0 & & (ustime ()-start) > timelimit) {timelimit_exit = 1; return;} / / if the percentage of failed primary keys in the sample is more than 25%, then continue the sampling deletion process} while (expired > REDIS_EXPIRELOOKUPS_PER_CRON/4);}} thank you for reading! This is the end of this article on "sample Analysis of the principle of redis_memcached failure". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.