Shulou(Shulou.com)06/02 Report--

This article introduces the knowledge of "what is the structure of Redis database and what is persistence respectively". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Redis database, persistent database

The Redis server stores all the databases in the db array of the server state redis.h/redisServer structure, each project is a redis.h/redisDb structure, and each redisDb structure represents a database.

Struct redisServer {/ /... / an array that holds all the databases in the server redisDb * db; / / the number of server databases int dbnum; / /.}

When the server is initialized, the program will decide how many databases to create according to the dbnum property of the server state, which is determined by the database option of the configuration item in redis.conf. The default value is 16.

Switch database using select + database number to switch, default is 0 database. For example: select 9.

Inside the server, the db property of the client state redisClient structure records the client's current target database, which is a pointer to the redisDb structure.

Typedef struct redisClient {/ /... / record the database redisDb * db; / /} redisClient; key space currently being used by the client.

Redis is a key-value database server, and each database has a redis.h/redisDb structure representation, in which the dict dictionary (hash table) holds all the key-value pairs in the database, and we call this dictionary the key space (key space).

Typedef struct redisDb {/ /... / database key space, save all the key-value pairs in the database dict * dict; / /.} redisDb

The key space corresponds directly to the database that the user sees:

The key of the key space is the key of the database, and each key is a string object.

The value of the key space is the value of the database, and each value can be one of string objects, list objects, hash table objects, collection objects, and ordered objects.

Illustration:

Redis > SET message hello world "OKredis > RPUSH alphabet ab c (integer) 3redis > HMSET book name" Redis Book "author" xxx "publisher" yyy "(integer) 3

EXPIRE

SETEX: this command can only be used for string objects, setting the expiration time when setting the value.

EXPIRE key ttl: used to set key survival time ttl seconds

PEXPIRE key ttl: used to set key survival time ttl milliseconds

EXPIREAT key timestamp: sets the timestamp expiration of the number of seconds specified by key in timestamp

PEXPIREAT key timestamp: sets the timestamp expiration of the number of milliseconds specified by key in timestamp

There are many commands, but the final execution is all converted to PEXPIRE commands at the bottom, that is, setting how many timestamps the key will expire.

The expiration time is not directly maintained in the key space dict dictionary. Think about it, after the key expires, we definitely need to clean up memory. At this time, if one traverses all the keys, it is too inefficient, so redis uses a new dictionary to save the expiration time. The expires dictionary of the redisDb structure holds the expiration time of all keys in the database. It is also a dict (hash table) structure

Typedef struct redisDb {/ /... / / expiration dictionary, which holds the expiration time of the key dict * expires; / /.} redisDb

Graphic illustration

Delete policy for expired keys

Since the key will expire, you certainly need to remove it to prevent it from taking up memory. So there may be several different strategies for how to delete expired keys, so let's take a look at it first.

Timing deletion: while setting the expiration time of the key, create a timer timer that allows the timer to delete the key immediately when the expiration time of the key is approaching.

Lazy deletion: leave expired keys alone, but every time you get a key from the key space, check to see if the key has expired, delete it if it expires, and return if it doesn't.

Delete regularly: every once in a while, the program checks the database and deletes the expired keys. It is up to the algorithm to decide how many expired keys to delete and how many databases to check.

Scheduled deletion

Timing deletion by using timers, this policy ensures that expired keys are deleted as soon as possible and frees up the memory occupied by expired keys.

But the disadvantage is that it is the most unfriendly to CPU time, in the case of more expired keys, delete keys this behavior may take up a considerable part of CPU time, in the case of memory is not tight but CPU time is very tight, it is undoubtedly a waste to use CPU time to delete expired keys that have nothing to do with the current task.

In addition, creating a timer requires time events in the Redis server, while the current time events are implemented without a linked list, so it is not efficient to deal with a large number of time events, and a large number of timers need to be created.

When there are too many expired keys, this approach is a little unrealistic.

Lazy deletion

The lazy delete policy is the best for CPU time, and the program will only check the expiration of the key when the key is removed, which ensures that deleting the expired key will only be done if necessary, and the target of the deletion is limited to the current key, and the policy will not waste any CPU time on deleting other unrelated expired keys.

But this approach also has the disadvantage that it is the least memory-friendly. Imagine that if a key has expired and the key is still stored in the database, then as long as we do not access the key, the key will never be deleted and the memory it occupies will not be released (prevent leverage, fix it, downmachine and other accidents are not included).

When using this strategy, if there are a lot of expired keys in the database and these parts are not accessed, then redis will be fine.

Delete periodically

From the point of view of the above regular deletion and lazy deletion, both methods have obvious defects when used alone.

Scheduled deletion takes up too much CPU time, affecting the response time and throughput of the server.

Want to delete too much memory, there is a danger of memory leakage.

The periodic deletion policy is, of course, a compromise between these two things, by deleting expired keys at regular intervals and limiting the length and frequency of deletions to reduce the impact of deletions on CPU time. By deleting regularly, you can effectively see the memory waste caused by expired keys.

So the question is, how long is the appropriate length of each execution and how much is the frequency? If the execution is too frequent or too long, will it degenerate into a regular deletion? If the execution is too little, or the time is too short, then there will also be a waste of memory.

Therefore, this kind of situation needs to be designated by experienced bosses according to the situation. Next, let's talk about the deletion strategy of Redis.

Redis's expired key deletion policy

The Redis server actually uses two strategies: lazy deletion and periodic deletion. By using it together, the server can strike a good balance between reasonable use of CPU time and avoiding wasting memory space.

In fact, it is easy to understand that when you delete the implementation on a regular basis, you also add expired verification when taking a value.

The influence of expired keys on RDB, AOF and replication

Because of the regular deletion policy, there must be cases where they are expired but not deleted yet. What impact does this have on Redis persistence and master-slave replication? Ha ha, of course, there is no impact of TM, there is an impact is bug.

RDB

If the RDB function is enabled

If it is master, when loading the RDB file, the program will check the keys saved in the file and load only those keys that have not expired.

In the case of slave: load everything, but it doesn't affect it, because when the master and slave servers synchronize, the slave server's database will be emptied.

AOF

When AOF mode persistence runs, if a key in the database has expired, it will not affect AOF. AOF will also record the key. When the key is deleted, the program will append a DEL command to the AOF file to record that the key has been deleted.

During AOF rewriting, similar to generating a RDB file, expired keys are not saved to the rewritten AOF file.

Copy

When running in replication mode:

After deleting an expired key, the master server explicitly sends a DEL command to all slave servers telling them to delete the expired key from the server.

When the read command sent by the client is executed from the server, it will not be deleted even if the expired key is encountered. Of course, if it expires, it will not be returned to the client because it will be judged.

The slave server deletes the expired key only after receiving a DEL command from the master service.

Through the master server to control the removal of expiration keys from the server, the data consistency of the master and slave servers can be ensured.

RDB persistence

There are two commands to generate the RDB file.

SAVE: blocks the server process, during which no command requests can be processed until the RDB file is created.

BGSAVE: derives the child process, which is responsible for creating the RDB file, and the main process continues to process command requests.

The loading of RDB files is performed automatically when the server starts, and all Redis does not have a special command for loading RDB files. As long as the server detects the existence of RDB files when the server is restarted, it will automatically load RDB files (if AOF is not open).

Look at the log:

32188 seconds32188:M M 25 Jun 2019 1834 Jun 01.999 # Server initialized32188:M 25 Jun 2019 18V 34R 01.999 * DB loaded from disk: 0.000 seconds32188:M 25 Jun 2019 18R 34R 01.999 * Ready to accept connections automatic interval storage

Redis allows users to automatically execute BGSAVE commands at regular intervals by configuring the server's save option.

You can set multiple save conditions, and the BGSAVE command will be executed as long as any one of them is met.

For example:

# at least one modification to the database within 900s, save 900s within 300s, at least 10 modifications to the database save 300 1m within 60 seconds, at least 10000 modifications to the database save 60 10000

The server redisServer maintains the dirty and lastsave properties, which are used to hold the change count and the last time the save was executed.

Struct redisServer {/ /... / modify counter long long dirty; / / the last time the save was performed time_t lastsave; / /...}

Each time the server successfully executes a modify command, the program reads the dirty counter and updates it.

In order to check whether the preservation condition is met, Redis's server periodic operation function serverCron is executed every 100ms by default. This function is used to maintain the running server. One of his tasks is to check whether the set preservation condition is met, and execute the BGSAVE command if it meets the requirement.

RDB file structure

REDIS: this section is 5 bytes long and holds 5 characters of "REDIS". With these 5 characters, the program can quickly check whether the loaded file is an RDB file when loading the file.

Db_version: 4 bytes long, whose value is an integer represented by a string, recording the version number of the RDB.

Database: this section contains 0 or any number of databases and key-value pairs in each database. If the database is empty, this part is also empty, with a length of 0 bytes.

EOF: a constant, 1 byte in length, that marks the end of the body content of the RDB file.

Check_sum: an 8-byte unsigned integer that holds a checksum, calculated by calculating the contents of the previous four parts, and used to check the RDB file for errors or corruption.

Databases part

The databases section of a RDB file can hold any number of non-empty databases.

For example, if databases 0 and 3 are not empty, the following RDB file will be created. Database 0 represents all key-value pair data in 0 good database, and database 3 represents all key-value pair data in database 3.

Each non-empty database database num can be saved as SELECTDB, db_number and key_value_pair in the RDB file.

SELECTDB: constant, 1 byte, and when the program encounters this value, it knows that it is about to read a database number.

Db_num: holds a database number, which can be 1 2 or 5 bytes, depending on the size of the number.

Key_value_pair: holds all the key-value pairs.

When the database number is read, the server invokes the SELECT command to switch the database so that the key-value pairs read later can be loaded into the correct database.

Post a picture every day:

Key_value_pair

This section holds all the key-value pairs, and if the key-value pair has an expiration time, then the expiration time will also be saved.

The key-value pair without expiration time is composed of TYPE, key and value in the RDB file.

The key-value pair with expiration time is composed of EXPIRETIME_MS, ms, TYPE, key and value in the RDB file.

Post a picture every day.

")

TYPE indicates the current type of key (such as REDIS_RDB_TYPE_SET) and determines how redis reads and interprets value data next.

Key is always a string object.

Value will vary according to the type of TYPE and the length of saved content, so this part will not be explained one by one. Go to Baidu if you like.

AOF persistence

The implementation of AOF persistence function can be divided into three steps: append, file writing and file synchronization (sync).

Append

When AOF persistence is turned on, after the server executes another write command, it is appended to the end of the aof_buf buffer of the server state in the format of the protocol.

Struct redisServer {/ /... / / AOF buffer sds aof_buf; / /...}

For example:

Redis > set mKey mValueOK

Then the server will convert it to the protocol format and append it to the end of the aof_buf buffer after being intellectually gentle.

Protocol format * 3\ r\ nroom3\ r\ nSET\ r\ nroom4\ r\ nmKey\ r\ nroom6\ r\ nmValue\ r\ nFile write and sync

The server process of Rdis is an event loop, in which the file event is responsible for receiving the command request from the client and sending the command reply to the client, while the time event is responsible for executing functions such as the serverCron function that need to be run regularly.

Because the server may execute write commands when handling file events, the contents of these commands are appended to the aof_buf buffer, so before the server ends an event loop each time, he will call the corresponding function (flushAppendOnlyFile) to consider whether the contents of the aof_buf buffer need to be written to and saved to the AOF file.

The behavior of the flushAppendOnlyFile function is determined by the value of the server configuration appendfsync option.

The appendfsync value function acts as always writes and synchronizes all the contents of the aof_buf buffer to the AOF file everysec [system default] writes the contents of the aof_buf buffer to the AOF file, and if the last time the AOF file has been synchronized for more than one second, then the AOF file is synchronized again, and this synchronization operation is performed by a thread that is responsible for writing all the contents of the aof_buf buffer to the AOF file. AOF files are not synchronized alone, and the appropriate synchronization is decided by the operating system.

AOF has a higher priority than RDB!

Because the AOF file contains all the write commands needed to rebuild the database state, the server can restore the server state simply by reading in and re-executing the write commands saved in the AOF file. Because redis's commands can only be executed in the client context, you need to use a pseudo client (fake client) here.

Flow chart:

AOF rewriting

Because AOF persistence records the state of the database by saving the write commands executed, with the increase of server running time, there will be more and more contents in the AOF file, the size of the file will become larger and larger, and the more time it takes to restore.

For example, some expired keys are written at first and then DEL later, or we use a lot of RPUSH commands to manipulate the list data of a key, and so on.

At this point, you need to rewrite the AOF file and replace the old one with the new one. File rewriting does not require any operations such as reading and analysis of existing AOF files, but is realized according to the current database state.

Of course, if lists, hash tables, collections. When the four ordered sets may have keys of multiple elements, the number will be checked first. if there are too many, they will be recorded in multiple commands instead of using a single command. The value of the REDIS_AOF_REWRITE_ITEMS_PRE_CMD constant (64) determines this number, that is, each instruction will write up to 64 elements, and the rest will continue to judge and decide how many instructions to write.

Background rewrite

The rewriting of AOF is also executed by subroutines so that the main program can continue to execute command requests. At this time, the new command received by the server may also modify the database, resulting in the inconsistency between the current database state and the database state saved by the rewritten AOF file.

So the Redis server sets an AOF rewrite buffer, which is used after the server creates a child process, and when a write command is executed, it sends the write command to both the AOF buffer and the AOF rewrite buffer.

In this way, the contents of the AOF buffer are periodically written and synchronized to the AOF file, and the processing of the existing AOF file continues as usual.

When the AOF rewrite is completed, the child process sends a signal to the parent process, and after the parent process receives the signal, it calls a signal function processor and performs the following work:

Writes all the contents of the AOF rewrite buffer to the new AOF file, which is the same as the current state of the new AOF file database.

Rename the new AOF file, atomic the existing AOF file, and complete the replacement of the old and new AOF file.

Daily process:

Time server process (parent process) child process T1SET K1 v1

T2SET k1 v2

T3SET k1 v3

T4 create child process, execute AOF file rewrite start AOF file rewrite T5SET K2 v2 execute rewrite operation T6SET K3 v3 execute rewrite operation T7SET K4 v4 complete AOF rewrite, send a signal to the parent process T8 receives the signal from the child process, append the command SET K2 v2 Magi set K3 set K4 v4 to the end of the new AOF file

T9 overwrites the old AOF file with the new AOF file

This is the end of the content of "what is the Redis database structure and persistence respectively". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.