Shulou(Shulou.com)06/01 Report--

Deeply explore the principle of Redis persistence

Redis is an in-memory database. In order to ensure the persistence of data, redis provides two persistence methods: RDB and AOF.

Redis is an in-memory database. In order to ensure data persistence, redis provides two persistence methods: RDB and AOF. Let's take a look at the implementation principles of these two persistence methods respectively.

RDB (default)

RDB is done through snapshots, and when certain conditions are met, redis will automatically persist the data in memory to disk.

The timing of the snapshot is triggered in accordance with the custom configured snapshot rules. (configured in redis.conf, described in more detail below) execute save or bgsave commands, execute flushall commands

Perform a master-slave copy operation (first time)

Schematic diagram

In the process of the snapshot, that is, in the process of generating the file, the original rdb file will not be modified until the snapshot is generated, and the old one will be replaced with the new one directly to ensure that the rdb file is complete at any time. We can back up redis data by backing up rdb files regularly. Rdb is a compressed binary file, which takes up less space and is convenient for transmission.

# Snapshot setting rules save {number of seconds} {how much data has changed}

For example: save 101: snapshot when at least one key is modified within 100s; save 2004Snapshot when at least 4 keys are modified within 200s.

The pros and cons of RDB: using RDB for persistence, when redis exits abruptly, the data after the last snapshot will be lost. However, you can set up automatic snapshots according to the combination to reduce data loss and ensure that it is within the range of acceptance. If the data is important, you can use the advantages of AOF: using RDB can maximize redis performance. In the snapshot process, we can see that the main process only needs to fork a child process, and all the rest of the work is done by the child process. The parent process does not need to perform any disk I / O operations. However, if the dataset is large, it can be time-consuming during the fork child process, which can cause redis to stop processing requests for a period of time. AOF mode

By default, redis does not turn on AOF (append only file). After starting AOF, every time redis receives a command to change redis data, it writes the command to the AOF file on the hard disk. Obviously, this process will degrade the performance of redis, but it is acceptable in most cases. At the same time, using a better hard disk can improve AOF performance.

AOF configuration method # enable appendonly parameter appendonly true# set AOF file location dir. / # set AOF file name. Default is appendonly.aofappendfilename appendonly.aof

Schematic diagram

RESP protocol

The Redis client communicates with the Redis server using the RESP protocol, which is specially designed for redis, but can also be used for other Cmurs projects.

Interval symbol:\ r\ n in linux,\ nsimple string begins with'+ 'error Errors, integer type Integer begins with' -', large string begins with array type Arrays with'$', and AOF principle is explained with'*'.

Redis persists the data by recording all write commands to the AOF file. The process of recording commands to an AOF file can be divided into three phases:

Command propagation cache append file write and save command propagation

Redis sends the completed command, command parameters, the number of command parameter boxes and other contents to the AOF program. When the redis client executes the command, it sends the protocol text to redis,redis through connection, receives the protocol text, selects the appropriate command function according to the content, converts the protocol text into Redis string object, and sends the command parameters to the AOF program after the execution of the command function.

Cache append

After receiving the command parameter, the AOF program converts it from the string object to the protocol content, and then appends the protocol content to the AOF cache. The AOF cache is in the aof_buf of the redisServer structure, and the new content is appended to the end of the aof_buf. Aof_buf maintains all protocol text that is not written to the AOF file.

File writing and saving

The contents of the AOF cache are written to the AOF file and kept to disk. When the server's regular function is executed, or the event handler is executed, the flushAppendOnlyFile function will be executed. There will be two processes.

WRITE: write the contents of AOF cache to the AOF file SAVE: call the fsync or fdatasync function to save the AOF file to disk

There are three save modes for AOF

AOF_FSYNC_NO: do not save. In this mode, each time the flushAppendOnlyFile function is called, write is executed and save is ignored. Save can only be triggered under the following three conditions: redis is turned off, aof is turned off, and the write cache of the system is flushed (maybe the cache is full, or periodic hold operations are performed). Performing save operations in these three cases will cause blocking of the redis main thread. AOF_FSYNC_EVERYSEC (default): save once per second. The save is executed once a second, but the save is done by the backstage child thread and does not cause the redis main thread to block. AOF_FSYNC_ALWAYS: save every command you execute. In this case, every command write and save executed is executed, and the save operation is completed by the main thread, resulting in redis blocking. It is not recommended because of its high security and poor performance. File Optimization of AOF

Redis can rewrite AOF files in the background (child processes) when the AOF file is too large. The new file after rewriting contains the minimum set of commands required to restore the current dataset. Override does not read or write to the AOF file, but for the current key in the database.

Set S1 1set S1 2set s1 3 after set s1 3 optimization

Principle of AOF file optimization

The rewriting of AOF is achieved through child processes, so the main thread continues to work, and it is possible to modify the new data, which may lead to inconsistency between the database data and the rewritten data. Redis solves this problem by adding an AOF rewrite cache. When the fork has a child process, the new command will not only be appended to the existing AOF file, but also a piece of data will be added to the cache.

Rewrite process analysis (need to ensure that slave operations are absolutely safe)

After Redis creates a new AOF file, it continues to add commands to the original AOF file, and even if the database goes down suddenly, the original AOF file and file contents will not be lost. When the new AOF file is created, it will directly replace the old one and add commands to the new AOF file.

When the child process is rewriting, the main process does the following

Process the request, continue adding new commands to the AOF file, and add commands to the AOF rewrite cache. Ensure consistent rows of data to avoid data loss. When the child process has finished rewriting, it will send a completion signal to the main process, and the main process will add the contents of the rewrite cache to the new AOF file, so that the contents of the new AOF file, database, and old AOF file are exactly the same. Then rename the new AOF file to overwrite the original AOF file.

In this way, the program completes the replacement of new and old AOF files. When the processing is complete, the main process continues to accept requests. In the whole rewriting process, only the last cache write and rename replacement will cause the main process to block, other times will not affect the normal operation of redis, and minimize the impact of AOF rewriting on the performance of redis.

Optimize trigger conditions

# the current AOF file will be rewritten when it exceeds the size of the AOF file when it was last rewritten. If it has not been rewritten, then the size of the AOF file at startup shall prevail as the auto-aof-rewrite-percentage 10 limits the smallest AOF file allowed to be rewritten, that is, there is no need to rewrite the optimized auto-aof-rewrite-min-size 64mb when the file is less than this value.

This is the end of the introduction about the principle of Redis persistence. I hope the above content can be used as a reference for everyone. If you like this article, you might as well share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.