Shulou(Shulou.com)05/31 Report--

Editor to share with you what the persistence mechanism in Redis is like. I hope you will get something after reading this article. Let's discuss it together.

Redis persistence mechanism

Why persistence?

If Redis accesses again and discovers that the data of Redis is empty, cache penetration will occur. More importantly, because the data in Redis is empty, there is no key that the client wants to access, which will cause a large number of requests to hit the database instantly, causing a cache avalanche (a small amount of key is penetration, a large number of key is avalanche).

At this point, the database may hang up. However, there is no guarantee that the redis will not go down, so you need to restore the contents of the redis quickly after the downtime. So you need to do a persistence. Persistence is used to recover data, not to store it.

RDB

RDB (Redis DataBase) is the default storage method of Redis, and the RDB mode is accomplished through snapshotting.

How to trigger the snapshot

Comply with custom configured snapshot rules

Save 9001 # means that at least one key is changed within 15 minutes (900s) before taking a snapshot.

Save 30010 # means that at least 10 keys are changed within 5 minutes (300 seconds) before taking a snapshot.

Save 60 10000 # means that if at least 10000 keys are changed in one minute, it will proceed quickly.

When the condition that there are at least M changes to the dataset within N seconds is met, the dataset is automatically saved once.

Execute the save or bgsave command

Execute the command save or bgsave to generate a dump.rdb file, and each command execution will take a snapshot of all redis memory into a new rdb file and overwrite the original rdb snapshot file.

Comparison of save and bgsave:

Whether the command savebgsaveIO type is synchronous and asynchronous whether blocking redis other commands (there will be temporary blocking when generating child processes to call fork functions) complexity O (n) O (n) advantages do not consume extra memory does not block client commands blocking client commands require Fork child processes, consume memory

Bgsave is used in the background to configure the automatic generation of rdb files.

Execute the flushall command

Flushall

Save the current Redis snapshot before emptying Redis

Perform a master-slave copy operation (first time)

The rdb file needs to be generated for the first master-slave replication, and the current Redis snapshot will be saved

RDB execution process

Process analysis

After the parent process forks the child process, continue to work.

The child process sends a signal to the parent process that it is complete, and the parent process updates the statistics.

The child process creates the RDB file, generates a temporary snapshot file according to the memory snapshot of the parent process, and then atomic replaces the original file. (RDB is always intact)

After the parent process fork, the bgsave command returns "Background saving started" information which no longer blocks the parent process and can respond to other commands.

The parent process executes the fork (calling the operating system function to copy the main process) to create the child process, in which the parent process is blocked and Redis cannot execute any commands from the client.

The Redis parent process first determines whether the child process of save or bgsave/bgrewriteaof (aof file rewrite command) is currently executing, and if so, the bgsave command returns directly.

RDB file structure

1. The header 5 bytes is fixed as a "REDIS" string.

2, 4-byte "RDB" version number (not Redis version number), currently 9, populated with 0009

3. Auxiliary fields in the form of key-value

4. Store the database number

5. Dictionary size

6. Expired key

7. Main data, stored in the form of key-value

8. End mark

9. Checksum, that is, to see if the file is damaged or modified

Advantages and disadvantages of RDB

Advantages

RDB is a binary compressed file that takes up a small space and is easy to transfer (to slaver)

The main process fork child process, can maximize Redis performance, the main process can not be too large, the main process blocking in the replication process

Shortcoming

If data integrity is not guaranteed, all data changed since the last snapshot will be lost.

AOF

AOF (append only file) is another way to persist Redis. Redis is not enabled by default. When AOF persistence is enabled, Redis records all the commands (and its parameters) (RESP) written to the database into the AOF file, so as to record the state of the database.

In this way, when Redis restarts, as long as you play back these commands in order, they will return to their original state. AOF will record the process, RDB will only care about the results

Implementation of AOF persistence

Configure redis.conf

# you can open the appendonly file by modifying the appendonly parameter in the appendonly yes# AOF configuration file. The location where the appendonly yes# AOF file is saved is the same as the location of the RDB file, both set by the dir parameter. Dir. / # the default file name is appendonly.aof, and you can modify appendfilename appendonly.aof through the appendfilename parameter

AOF principle

Redis commands are stored in AOF files, and the whole process of synchronizing commands to AOF files can be divided into three stages:

Command propagation: Redis sends information such as the finished command, the parameters of the command, and the number of parameters of the command to the AOF program.

Cache append: according to the command data received, the AOF program converts the command into the format of the network communication protocol, and then appends the protocol content to the AOF cache of the server.

File write and save: the contents of the AOF cache are written to the end of the AOF file, and if the set AOF save conditions are met, the fsync function or fdatasync function will be called to actually save the written content to disk.

Command propagation

When a Redis client needs to execute a command, it sends the protocol text to the Redis server over a network connection. After receiving the request from the client, the server selects the appropriate command function according to the content of the protocol text and converts each parameter from the string text to the Redis string object (StringObject). Whenever the command function is successfully executed, the command parameters are propagated to the AOF program.

Cache append

When the command is propagated to the AOF program, the program converts the command from the string object back to the original protocol text according to the command and its parameters. After the protocol text is generated, it is appended to the end of the aof_buf of the redis.h/redisServer structure.

The redisServer structure maintains the state of the Redis server, while the aof_ buf domain holds all the protocol text (RESP) waiting to be written to the AOF file.

File writing and saving

Whenever the server general task function is executed, or the event handler is executed, the aof.c/flushAppendOnlyFile function is called, which performs the following two tasks:

WRITE: writes the cache in aof_buf to the AOF file based on conditions.

SAVE: call the fsync or fdatasync function based on the condition to save the AOF file to disk.

AOF save mode

Redis currently supports three AOF save modes, which are:

AOF_FSYNC_NO: do not save.

AOF_FSYNC_EVERYSEC: save every second. (default)

AOF_FSYNC_ALWAYS: save every command you execute. (not recommended)

AOF_FSYNC_NO

Never fsync, the data is handed over to the operating system for processing. Faster and less secure options.

SAVE will only be executed in any of the following situations:

Redis is shut down.

The AOF function is turned off

The system's write cache is flushed (perhaps because the cache is full, or periodic save operations are performed)

The SAVE operation in all three cases will cause the Redis main process to block.

AOF_FSYNC_EVERYSEC

In principle, SAVE executes every second, because the SAVE operation is called by the backstage child thread (fork), so it does not cause the server main process to block, and only 1 second of data is lost in the event of a failure.

AOF_FSYNC_ALWAYS

After each execution of a command, WRITE and SAVE are executed. Fsync is executed every time a new command is appended to the AOF file, which is very slow and secure.

Because SAVE is executed by the Redis main process, during SAVE execution, the main process is blocked and cannot accept command requests.

The influence of AOF save mode on performance and security

Comparison of three modes

AOF rewriting

The changing process of data recorded by AOF is getting larger and larger, so it is necessary to rewrite "slimming"

Redis can automatically rewrite AOF in the background (form child process) when the AOF becomes too large.

The new rewritten AOF file contains the minimum set of commands required to restore the current dataset.

The so-called "rewrite" is actually an ambiguous word. In fact, AOF rewriting does not require any writing or reading to the original AOF file, it is aimed at the current value of the key in the database.

Give examples to illustrate

Set s1 11set s1 22set s1 33lpush list1 1 2 3lpush list1 4 5 6

After AOF rewriting

Set s1 33lpush list1 1 2 3 4 5 6

Redis did not want AOF rewriting to prevent the server from processing requests, so Redis decided to put the AOF rewriting program into a (background) child process to execute

1. During the AOF rewrite of the child process, the main process can continue to process command requests.

2. The child process has a copy of the data of the main process. Using the child process instead of the thread can ensure the security of the data without locking.

However, there is a problem with using child processes: because the main process needs to continue to process commands while the child process is rewriting AOF, and the new command may modify the existing data, which will make the data in the current database inconsistent with the data in the rewritten AOF file.

To solve this problem, Redis added an AOF rewrite cache, which is enabled after the fork exit child process. After receiving a new write command, the Redi main process will append the protocol contents of the write command to the existing AOF file as well as to the cache.

Rewrite process analysis

During the process of creating a new AOF file, Redis will continue to append commands to the existing AOF file, and even if a shutdown occurs during the rewrite process, the existing AOF file will not be lost. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and starts appending the new AOF file.

When a child process is performing an AOF rewrite, the main process needs to perform the following three tasks:

Process the command request.

Appends the write command to the existing AOF file.

Append write commands to the AOF rewrite cache

. This ensures that existing AOF functions will continue to execute, and that even if downtime occurs during AOF rewriting, there will be no data loss. All commands that make changes to the database are recorded in the AOF rewrite cache.

When the child process completes the AOF rewrite, it sends a completion signal to the parent process, and after receiving the completion signal, the parent process calls a signal handling function and completes the following work:

Writes all the contents of the AOF rewrite cache to the new AOF file.

Rename the new AOF file to overwrite the original AOF file.

Commands in the + AOF rewrite process in the Redis database-> New AOF file-- > overwrite the old one. When step 1 is finished, the status of the existing AOF file, the new AOF file, and the database is exactly the same.

After the execution of step 2, the program completes the alternation of the old and new AOF files. After the signal processing function is executed, the main process can continue to accept command requests as usual.

. In the whole AOF background rewrite process, only the final write cache and rename operation will block the main process, and at other times, the AOF background rewrite will not block the main process, which minimizes the impact of AOF rewriting on performance.

AOF rewrite trigger mode

1. Configuration trigger

# indicates that the current aof file size will be rewritten when it exceeds how many percent of the previous aof file size. If it has not been rewritten before, the auto-aof-rewrite-percentage 10 limit allows you to rewrite the minimum aof file size based on the startup aof file size, that is, when the file size is less than 64mb, there is no need to optimize auto-aof-rewrite-min-size 64mb

2. Execute the bgrewriteaof command

127.0.0.1 purl 6379 > bgrewriteaof'Backgroundappendonlyfilerewritingstarted

AOF rewrite summary

Mixed persistence

RDB and AOF have their own advantages and disadvantages, and Redis 4.0 has begun to support mixed persistence of rdb and aof.

If you turn on mixed persistence, aofrewrite will write the contents of rdb directly to the beginning of the aof file.

Head of RDB + body of AOF-> appendonly.aof

Enable mixed persistence

Aof-use-rdb-preambleyes

Loading and data Restoration of AOF File

If hybrid persistence is enabled, when AOF rewrites, it no longer simply converts memory data into RESP commands to write to AOF files, but does RDB snapshot processing on memory before rewriting this moment, and writes the contents of RDB snapshots and incremental AOF commands to modify memory data together, writing to a new AOF file. The new file is not called appendonly.aof at first, but will not be renamed until the new AOF file is rewritten. Overwrite the original AOF file and complete the replacement of the old and new AOF files.

Therefore, when Redis is restarted, the contents of RDB can be loaded first, and then the incremental AOF logs can be replayed, which can completely replace the previous AOF full file replay, so the restart efficiency has been greatly improved.

1. Create a pseudo client without network connection (fake client)

Because the command of Redis can only be executed in the context of the client, and the command used when loading the AOF file comes directly from the AOF file rather than the network connection, the server uses a pseudo client without a network connection to execute the write command saved by the AOF file. The effect of the pseudo client executing the command is exactly the same as that of the client with a network connection.

2. Analyze and read a write command from the AOF file

3. Use the pseudo client to execute the read write command

4. Continue to perform steps 2 and 3 until all write commands in the AOF file are processed

Redis data backup strategy

1. Write crontab scheduled scheduling scripts, copy a backup of rdb or aof to a directory every hour, and only keep the backup for the last 48 hours

two。 Keep a copy of the same day's data backup to a directory every day, and you can keep the backup of the last month.

3. Every time copy backup, delete the backup that is too old

4. Copy a copy of the backup on the current machine to another machine every night in case the machine is damaged

Comparison between RDB and AOF

1. RDB stores snapshots of data at a certain time, using binary compression storage, AOF storage operation commands, and text storage (hybrid)

2. High performance of RDB and low performance of AOF.

3. RDB will lose all data changed after the last snapshot when the trigger state is configured. If AOF is set to save once per second, the data will be lost for up to 2 seconds.

4. Redis runs in master server mode. RDB does not save expired key-value pairs. Redis runs in slave server mode. RDB saves expired key-value pairs. When the master server synchronizes with the slave server, it clears the expired key-value pairs. When AOF writes to a file, an del command is appended to the expired key, and the expired key and del commands are ignored when performing an AOF rewrite.

After reading this article, I believe you have a certain understanding of "what is the persistence mechanism in Redis". If you want to know more about it, you are welcome to follow the industry information channel. Thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.