Shulou(Shulou.com)05/31 Report--

The main content of this article is to explain "what is the method of database redis Db and key expiration deletion". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what is the method of database redis Db and key expiration deletion"?

one。 Database

The database of Redis uses dictionary as the underlying implementation, and the addition, deletion, query and modification of the database are all built on the operation of the dictionary.

The redis server stores all databases in the db array (which should be a linked list) of the server state structure redisServer (redis.h/redisServer):

Struct redisServer {/ /.. / / database array, which holds all the databases in the server redisDb * db; / /..}

When initializing the server, the program determines how many databases should be created based on the dbnum property of the server state:

Struct redisServer {/ /.. / / the number of databases in the server int dbnum; / /..}

The value of the dbnum property is determined by the database option configured by the server, with a default value of 16

II. Principle of switching databases

Each Redis client has its own target database, and whenever the client executes read and write commands to the database, the target database becomes the operating object of these commands.

127.0.1 set msg 'Hello world'OK127.0.0.1:6379 > get msg "Hello world" 127.0.0.1 get msg 6379 > select 2OK127.0.0.1:6379 [2] > get msg (nil) 127.0.0.1 get msg 6379 [2] >

Inside the server, the db property of the client state redisClient structure (redis.h/redisClient) records the client's current target database, which is a pointer to the redisDb structure (redis.h/redisDb):

Typedef struct redisClient {/ /.. / / the database redisDb * db; / /..} redisClient currently being used by the client

The redisClient.db pointer points to an element in the redisServer.db array, and the element being pointed to is the current client's target database.

We can switch databases by modifying the redisClient pointer so that it points to different databases in the server-that's how the select command works.

Implementation code:

Int selectDb (redisClient * c, int id) {/ / ensure that id is within the correct range of if (id

< 0 || id >

= server.dbnum) return REDIS_ERR; / / switch database (update pointer) c-> db = & server.db [id]; return REDIS_OK;} 3. Database key space 1, database structure (we only analyze key space and key expiration time) typedef struct redisDb {/ / database key space, holds all key value pairs in the database dict * dict / * The keyspace for this DB * / / the expiration time of the key, the key of the dictionary is the key, and the value of the dictionary is the expired event UNIX timestamp dict * expires; / * Timeout of keys with a timeout set * / / database number int id / * Database ID * / / the average TTL of keys in the database, statistics long long avg_ttl; / * Average TTL, just for stats * / /.} redisDb

The figure above is an example of RedisDb, which holds five key-value pairs, sRedis,INums,hBooks,SortNum and sNums, each with its own value object, and three of them set the expiration time, and the current database is the server's database No. 0. Now, let's analyze the database structure from the perspective of source code:

We know that Redis is a key-value database server, and every database in the server is a redis.h/redisDb structure, in which the dict dictionary in the structure stores all the key-value pairs in the database, so we turn this dictionary into a key space.

The data of Redis database exists in the form of key-value pairs, which makes full use of the characteristics of efficient dictionary index.

A, the key of the key space is the key in the database, which is usually a string object.

B. the value of the key space is the value in the database and can be one of five types of objects (string, list, hash, collection, and ordered collection).

After the analysis of the key spatial structure of the database, let's first look at the initialization of the database.

2. Initialization of key space

In redis.c, we can find the initialization of the key space:

/ / create and initialize the database structure for (j = 0; j)

< server.dbnum; j++) { // 创建每个数据库的键空间 server.db[j].dict = dictCreate(&dbDictType,NULL); // ... // 设定当前数据库的编号 server.db[j].id = j;} 初始化之后就是对键空间的操作了。 3、键空间的操作 我先把一些常见的键空间操作函数列出来： // 从数据库中取出键key的值对象，若不存在就返回NULLrobj *lookupKey(redisDb *db, robj *key);/* 先删除过期键，以读操作的方式从数据库中取出指定键对应的值对象 * 并根据是否成功找到值，更新服务器的命中或不命中信息, * 如不存在则返回NULL，底层调用lookupKey函数 */robj *lookupKeyRead(redisDb *db, robj *key);/* 先删除过期键，以写操作的方式从数据库中取出指定键对应的值对象 * 如不存在则返回NULL，底层调用lookupKey函数， * 不会更新服务器的命中或不命中信息 */robj *lookupKeyWrite(redisDb *db, robj *key);/* 先删除过期键，以读操作的方式从数据库中取出指定键对应的值对象 * 如不存在则返回NULL，底层调用lookupKeyRead函数 * 此操作需要向客户端回复 */robj *lookupKeyReadOrReply(redisClient *c, robj *key, robj *reply);/* 先删除过期键，以写操作的方式从数据库中取出指定键对应的值对象 * 如不存在则返回NULL，底层调用lookupKeyWrite函数 * 此操作需要向客户端回复 */robj *lookupKeyWriteOrReply(redisClient *c, robj *key, robj *reply);/* 添加元素到指定数据库 */void dbAdd(redisDb *db, robj *key, robj *val);/* 重写指定键的值 */void dbOverwrite(redisDb *db, robj *key, robj *val);/* 设定指定键的值 */void setKey(redisDb *db, robj *key, robj *val);/* 判断指定键是否存在 */int dbExists(redisDb *db, robj *key);/* 随机返回数据库中的键 */robj *dbRandomKey(redisDb *db);/* 删除指定键 */int dbDelete(redisDb *db, robj *key);/* 清空所有数据库，返回键值对的个数 */long long emptyDb(void(callback)(void*)); 下面我选取几个比较典型的操作函数分析一下： 查找键值对函数-lookupKey robj *lookupKey(redisDb *db, robj *key) { // 查找键空间 dictEntry *de = dictFind(db->

Dict,key- > ptr); / / if (de) {/ / take out the value corresponding to the key robj * val = dictGetVal (de); / / update time information if (server.rdb_child_pid = =-1 & & server.aof_child_pid = =-1) val- > lru = LRU_CLOCK (); / / return value return val } else {/ / Node does not exist return NULL;}}

Add key-value pair-dbAdd

Add a key-value pair to the function that we often use. The underlying function is implemented by the dbAdd () function. The passed parameters are the database, key object and value object to be added. The source code is as follows:

Void dbAdd (redisDb * db, robj * key, robj * val) {/ / copy the key name sds copy = sdsdup (key- > ptr); / / try to add the key value pair int retval = dictAdd (db- > dict, copy, val); / / stop redisAssertWithInfo (NULL,key,retval = = REDIS_OK) if the key already exists / / if cluster mode is enabled, save the key to if (server.cluster_enabled) slotToKeyAdd (key) in the slot;}

All right, this is the end of the analysis of the key space operation function. Other functions (in the file db.c) can be analyzed by yourself. if you have any questions, you can reply, and we can discuss it together!

Fourth, the expired key operation of the database

As we mentioned earlier, there is an expires pointer in the redisDb structure (see the figure above), which points to a dictionary structure that holds the expiration time of all keys, which is called the expiration dictionary.

Initialization of expired dictionaries:

/ / create and initialize the database structure

For (j = 0; j)

< server.dbnum; j++) { // 创建每个数据库的过期时间字典 server.db[j].expires = dictCreate(&keyptrDictType,NULL); // 设定当前数据库的编号 server.db[j].id = j; // .. } a、过期字典的键是一个指针，指向键空间中的某一个键对象（就是某一个数据库键）； b、过期字典的值是一个long long类型的整数，这个整数保存了键所指向的数据库键的时间戳-一个毫秒精度的unix时间戳。 下面我们就来分析过期键的处理函数： 1、过期键处理函数 设置键的过期时间-setExpire() /* * 将键 key 的过期时间设为 when */void setExpire(redisDb *db, robj *key, long long when) { dictEntry *kde, *de; // 从键空间中取出键key kde = dictFind(db->

Dict,key- > ptr); / / if the key cannot be found in the key space, redisAssertWithInfo (NULL,key,kde! = NULL) is reported; / / add the key de = dictReplaceRaw (db- > expires,dictGetKey (kde)) to the expiration dictionary; / / set the expiration time of the key / / here the expiration time is saved directly using the integer value, not the String object dictSetSignedIntegerVal (de,when) encoded with INT } get the expiration time of the key-getExpire () long long getExpire (redisDb * db, robj * key) {dictEntry * de; / / if the expired key does not exist, return if (dictSize (db- > expires) = = 0 | (de = dictFind (db- > expires,key- > ptr)) = = NULL) return-1; redisAssertWithInfo (NULL,key,dictFind (db- > dict,key- > ptr)! = NULL) / / return expiration time return dictGetSignedIntegerVal (de);} delete key expiration time-removeExpire () / / remove key key expiration time int removeExpire (redisDb * db, robj * key) {/ / make sure the key has an expiration time redisAssertWithInfo (NULL,key,dictFind (db- > dict,key- > ptr)! = NULL); / / delete expiration time return dictDelete (db- > expires,key- > ptr) = = DICT_OK } 2. Expired key deletion policy

From the previous introduction, we should all know that the expiration time of the database key is saved in the expired dictionary, so if a key expires, when is the expired key deleted? Now let's take a look at the deletion policy of redis's expired keys:

A, timing deletion: while setting the expiration time of the key, create a timer and delete the key when the timing ends.

B, lazy deletion: let the key expire regardless of the expiration time. When accessing the key, determine whether the expiration time of the key has arrived. If the expiration time has expired, delete the key.

C, regular deletion: every once in a while, the keys in the database are traversed to delete expired keys.

Among them, regular deletion can delete expired keys in the database in time and release the memory occupied by expired keys, but it opens a timer for each key that sets the expiration time, which increases the load of cpu, which will affect the response time and throughput of the server.

Lazy deletion effectively overcomes the impact of regular deletion on CPU, but if an expired key is not accessed for a long time, and if there are a large number of such expired keys, it is bound to take up a lot of memory space, resulting in excessive memory consumption.

Regular deletion can be regarded as a compromise between the two strategies mentioned above. Set a timer to traverse the database every once in a while and delete the expiration key, which effectively alleviates the occupation of CPU by timing deletion and the memory occupation by lazy deletion.

In practical application, Redis adopts two strategies of lazy deletion and regular deletion to deal with expired keys. The lookupKeyWrite and other functions mentioned above make use of the lazy deletion strategy, while the timed delete policy performs deletion operations according to the server's routine handler serverCron, which is called every 100ms.

Lazy delete function-expireIfNeeded ()

The source code is as follows

/ * check whether the key has expired, and if so, delete it from the database * and write the delete command to the AOF file and the subsidiary node (master-slave replication and AOF persistence related) * return 0 indicates that the key has not expired Or no expiration time is set * return 1 means that the key is deleted because it expires * / int expireIfNeeded (redisDb * db, robj * key) {/ / get the expiration time of the key mstime_t when = getExpire (db,key) Mstime_t now; / / this key does not set the expiration time if (when

< 0) return 0; // 服务器正在加载数据的时候，不要处理 if (server.loading) return 0; // lua脚本相关 now = server.lua_caller ? server.lua_time_start : mstime(); // 主从复制相关，附属节点不主动删除key if (server.masterhost != NULL) return now >

When; / / the key has not expired if (now id); / / remove the key from the database return dbDelete (db,key);}

Delete policies on a regular basis

The periodic deletion policy of expired keys is implemented by the redis.c/activeExpireCycle () function. When the server operates redis.c/serverCron () periodically (every 100ms), the activeExpireCycle () function is called to traverse each database in the server several times, randomly check the expiration time of some keys from the expires dictionary in the database, and delete the expired keys.

Deleting expired keys is performed by the activeExpireCycleTryExpire function (called by activeExpireCycle ()), with the following source code:

/ * check the expiration time of the key, such as delete it directly * / int activeExpireCycleTryExpire (redisDb * db, dictEntry * de, long long now) {/ / get the expiration time long long t = dictGetSignedIntegerVal (de); if (now > t) {/ / execute until this instruction expires / / create a copy of the key sds key = dictGetKey (de); robj * keyobj = createStringObject (key,sdslen (key)) / / propagate the delete command to AOF and satellite node propagateExpire (db,keyobj); / / delete the key dbDelete (db,keyobj) in the database; / / send event notification notifyKeyspaceEvent (NOTIFY_EXPIRED, "expired", keyobj,db- > id); / / the reference count of temporary key objects minus 1 decrRefCount (keyobj) / / the expired key count of the server plus 1 / / this parameter affects the number of databases processed each time server.stat_expiredkeys++; return 1;} else {return 0;}} so far, I believe you have a deeper understanding of "database redis Db and key expiration deletion method". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.