Shulou(Shulou.com)06/02 Report--

This article focuses on "introduction to the persistence mechanism of RDB and AOF in Redis". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn the introduction of RDB and AOF persistence mechanism in Redis.

Redis data persistence

Redis as an in-memory database, the data is stored in memory, so once the Redis server process exits, the data in the server will disappear. To solve this problem, Redis provides a persistence mechanism, that is, to save the data in memory to disk to avoid accidental data loss.

Redis provides two persistence schemes: RDB persistence and AOF persistence, one is snapshot, the other is similar to log append

RDB snapshot persistence

RDB persistence is done by taking a snapshot, that is, writing a snapshot of a dataset in memory to disk within a specified time interval. After creating a snapshot, users can back up the snapshot, copy the snapshot to another server to create a server copy of the same data, or restore the data after restarting the server. RDB is the default persistence method for Redis

Snapshot persistence

RDB persistence produces a RDB file, which is a compressed binary file that allows you to restore the database state at the time of the snapshot, that is, the server data when the RDB file was generated. The RDB file defaults to the dump.rdb under the current working directory, and the file name and location of the RDB can be set according to the dbfilename and dir in the configuration file.

# set the file name of dump dbfilename dump.rdb# working directory # for example, the above dbfilename only specifies the file name, # but it will be written to this directory. This configuration item must be a directory, not a file name. Dir. /

The time to trigger the snapshot

Execute save and bgsave commands

Profile sets save rules and automatically executes bgsave commands at intervals

During master-slave replication, the master database data is replicated in full from the master database, and the master database executes bgsave.

Execute the flushall command to empty the server data

When you execute the shutdown command to close Redis, the save command is executed

Save and bgsave commands

By executing the save and bgsave commands, you can manually trigger the snapshot and generate the RDB file. The difference between the two is as follows

Using the save command blocks the Redis server process, which cannot process any command requests until the RDB file is created

127.0.0.1 purl 6379 > saveOK

Unlike using the bgsave command, the basave command will fork a child process, which will then be responsible for creating the RDB file, while the server process will continue to process command requests

127.0.0.1 purl 6379 > bgsaveBackground saving started

Fork () is a function provided by the operating system to create a copy of the current process as a child process

Fork a child process that first writes the dataset to a temporary file. After the write is successful, it replaces the previous RDB file and stores it with binary compression, which ensures that the RDB file always stores complete persistent content.

Automatic interval trigger

Set save rules in the configuration file to automatically execute bgsave commands at intervals

# # SNAPSHOTTING # # Save the DB on disk:## save # # Will save the DB if both the given number of seconds and the given# number of write operations against the DB occurred.## In the example below the behaviour will be to save : # after 900sec (15 min) if at least 1 key changed# after 300 sec (5 min) if at least 10 keys changed# after 60 sec if at least 10000 keys changed## Note: you can disable saving completely by commenting out all "save" lines.## It is also possible to remove all the previously configured save# points by adding a save directive with a single empty string argument# like in the following example:## save "save 900 1save 300 10save 60 10000

Save means that if there are at least changes changes within seconds seconds, the gbsave command will be triggered automatically.

Save 9001 when the time is up to 900s, if at least one key changes, the bgsave command is automatically triggered to create a snapshot

Save 30010 when the time is up to 300 seconds, if at least 10 key changes, the bgsave command is automatically triggered to create a snapshot

Save 60 10000 when the time is 60 seconds, if at least 10000 key changes, the bgsave command is automatically triggered to create a snapshot

AOF persistence

In addition to RDB persistence, Redis also provides AOF (Append Only File) persistence, which writes executed write commands to the end of the AOF file to record changes in data. By default, Redis does not enable AOF persistence. After it is enabled, every command that changes Redis data will be appended to the AOF file, which will reduce the performance of Redis, but in most cases this effect is acceptable. In addition, using a faster hard disk can improve the performance of AOF.

You can enable AOF persistence by configuring the redis.conf file. The configuration for AOF is as follows

# appendonly parameter enables AOF persistence appendonly no# AOF persistence file name. By default, appendonly.aofappendfilename "appendonly.aof" # AOF file is saved in the same location as the RDB file. The dir. / # synchronization policy is set through the dir parameter. # the implementation of whether to synchronize no-appendfsync-on-rewrite no# rewriting during appendfsync alwaysappendfsync everysec# appendfsync no# aof rewriting triggers configuration auto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mb# loading aof error how to handle aof-load-truncated yes# file rewriting policy aof-rewrite-incremental-fsync yesAOF

AOF needs to record every write command of Redis. The steps are: command appending (append), file writing (write), and file synchronization (sync).

Command append (append)

When the AOF persistence function is enabled, every time the server executes a write command, it appends the command to the end of the aof_buf cache in a protocol format instead of writing to the file directly, avoiding writing the command directly to the hard disk every time, thus reducing the number of IO on the hard disk

File write (write) and file synchronization (sync)

Redis provides several strategies for when to write and save the contents of the aof_buf buffer in the AOF file

Appendfsync always: writes and synchronizes all the contents of the aof_buf buffer to the AOF file, and each write command is written to disk synchronously

Appendfsync everysec: writes the contents of the aof_buf cache to the AOF file and synchronizes them once per second. A thread is responsible for this operation.

Appendfsync no: writes the contents of the aof_buf cache to the AOF file, and when to synchronize is decided by the operating system

The default configuration of the appendfsync option is everysec, that is, synchronization is performed once a second

The synchronization strategy of AOF is related to the write function and fsync function of the operating system, which is described in "Redis Design and implementation"

In order to improve the file writing efficiency, in the modern operating system, when the user calls the write function to write some data to the file, the operating system usually temporarily stores the data in a memory buffer. When the buffer space is filled up or exceeds the specified time limit, the buffer data is actually written to disk.

Although this operation improves efficiency, it also poses a security problem for data writing: if the computer is down, the data in the memory buffer will be lost. For this reason, the system provides fsync and fdatasync synchronization functions, which can force the operating system to write the data in the buffer to the hard disk immediately, so as to ensure the security of writing data.

As we know from the above introduction, the operating system does not necessarily synchronize the data we write to disk immediately, so Redis provides the option configuration of appendfsync. When this option is always, the data security is the highest, but a large number of writes are made to the disk, and the speed of Redis processing commands is limited by disk performance, while the appendfsync everysec option takes into account both data security and write performance, synchronizing AOF files at a frequency of once per second, even if the system crashes, the data generated within one second will be lost at most If it is the appendfsync no option, Redis will not synchronize the AOF files, but the operating system will decide when to synchronize, which will not affect the performance of Redis, but may lose an indefinite amount of data if the system crashes

AOF rewrite (rewrite)

Before we look at AOF rewriting, let's take a look at what is stored in the AOF file and perform two write operations

127.0.0.1 set 6379 > set S1 helloOK127.0.0.1:6379 > set S2 worldOK

Then we open the appendonly.aof file and see the following

* 3 $3set$2s1 $5hello*3 $3set$2s2 $5world

The command format is Redis Serialization Protocol (RESP). * 3 means that the command has three parameters, and $3 indicates that the length of the parameter is 3

Looking at the contents of the AOP file above, we should be able to imagine that with the passage of time, more and more write commands will be executed by Redis, and the AOF file will become larger and larger. Too large AOF files may affect the Redis server, and the longer it takes to use AOF files to restore data.

Over time, there are usually some redundant commands in the AOF file, such as commands with out-of-date data, invalid commands (duplicate settings, deletions), and multiple commands that can be merged into one command (batch command). So AOF files have room for reduction and compression.

The purpose of AOF rewriting is to reduce the size of AOF files, but it is worth noting that AOF file rewriting does not require any reading, sharing and writing to existing AOF files, but by reading the current database state of the server.

File rewriting can be divided into manual trigger and automatic trigger, which manually triggers the execution of bgrewriteaof commands, which are similar to those when bgsave triggers snapshots, in which a child process is first fork to do specific work.

127.0.0.1 purl 6379 > bgrewriteaofBackground append only file rewriting started

Automatic trigger automatically executes bgrewriteaof commands based on auto-aof-rewrite-percentage and auto-aof-rewrite-min-size 64mb configuration

# indicates that when the size of the AOF file is larger than 64MB, and the volume of the AOF file is twice as large as that after the last rewrite, the command auto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mb will be executed.

Let's take a look at the process of executing the bgrewriteaof command and rewriting.

Rewriting will have a lot of writes, so the server process will fork a child process to create a new AOF file

During the rewrite, the server process continues to process command requests, and if there is a written command, it will append to the aof_buf as well as to the aof_rewrite_bufAOF rewrite buffer

When the child process completes the rewrite, it will give a signal to the parent process, and then the parent process will write the contents of the AOF rewrite buffer into the new AOF temporary file, and then rename the new AOF file to complete the replacement, which can ensure the consistency of the new AOF file with the current database data.

Data recovery

Redis4.0 starts to support mixed persistence of RDB and AOF (can be enabled through the configuration item aof-use-rdb-preamble)

If the redis process is dead, then restart the redis process and recover the data directly based on the AOF log file

If the machine where the redis process is located is down, after restarting the machine, try to restart the redis process and try to recover the data directly based on the AOF log file. If the AOF file is damaged, use the redis-check-aof fix command to repair it.

If there is no AOF file, the RDB file will be loaded

If redis's current latest AOF and RDB files are lost / corrupted, you can try to recover data based on one of the latest RDB data copies currently on that machine

RDB vs AOF

RDB persistence and AOF persistence are introduced above, so let's take a look at their respective advantages and disadvantages and how to choose a persistence scheme.

Advantages and disadvantages of RDB and AOF

With regard to the advantages and disadvantages of RDB and AOF, https://redis.io/topics/persistence is also explained in more detail on the official website.

RDB

Advantages:

RDB snapshot is a very compact compressed file that holds a dataset at a certain point in time, which is suitable for data backup and disaster recovery.

The performance of Redis can be maximized. In saving RDB files, the server process only needs to fork one child process to complete the creation of RDB files, and the parent process does not need to do IO operations.

Recovery of large datasets is faster than AOF

Disadvantages:

The data security of RDB is not as secure as that of AOF, and the process of saving the entire dataset is more onerous. According to the configuration, it may take a few minutes to take a snapshot. If the server goes down, then several minutes of data may be lost.

When the Redis dataset is large, it takes more CPU and time for the child process of fork to complete the snapshot.

AOF

Advantages:

More complete data, higher security, and data loss in seconds (depending on the fsync policy, up to 1 second of data loss in the case of everysec)

The AOF file is an append-only log file, and the write operation is saved in the format of Redis protocol, and the content is readable, so it is suitable for erroneous deletion and emergency recovery.

Disadvantages:

For the same dataset, the volume of the AOF file is larger than that of the RDB file, and the data recovery is slow.

Depending on the fsync strategy used, AOF may be slower than RDB. However, in general, the performance of fsync per second is still very high.

How to choose RDB and AOF

If the data is less sensitive and can be regenerated from somewhere else, you can turn off persistence

If the data is important, you don't want to get it from anywhere else, and you can withstand a few minutes of data loss, such as caching, you can only use RDB.

If it is used as an in-memory database, to use Redis persistence, it is recommended that both RDB and AOF are enabled, or bgsave is executed regularly to make snapshot backups. RDB is more suitable for data backup, and AOF can ensure that the data is not lost.

At this point, I believe you have a deeper understanding of the "introduction to RDB and AOF persistence mechanisms in Redis". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.