Shulou(Shulou.com)05/31 Report--

Editor to share with you the example analysis of AOF persistence in Redis, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

1. Introduction to AOF

One of the persistence methods of Redis RDB records the state of the database by saving key-value pairs in the database. Another persistence method, AOF, records the state of the database by saving write commands executed by the Redis server.

For example, for the following command:

The method of RDB persistence is to save the three key-value pairs of str1,str2,str3 to the RDB file, while AOF persistence is to save the three set,sadd,lpush commands executed into the AOF file.

2. AOF configuration

In the redis.conf configuration file, under APPEND ONLY MODE:

①, appendonly: the default value is no, which means that redis defaults to rdb persistence. If you want to enable AOF persistence, you need to change appendonly to yes.

②, appendfilename: aof file name. Default is "appendonly.aof".

Configuration of ③ and appendfsync:aof persistence policies

No means that the fsync is not executed, and the operating system ensures that the data is synchronized to disk, which is the fastest, but not very secure.

Always means that fsync is executed every time a write is written to ensure that the data is synchronized to disk, which is very inefficient

Everysec indicates that fsync is executed every second, which may result in the loss of the 1s of data. Everysec is usually chosen to strike a balance between security and efficiency.

④, no-appendfsync-on-rewrite: a large number of IO will be executed when aof is rewritten or written to the rdb file. For the aof mode of everysec and always, the execution of fsync will block for too long, and the no-appendfsync-on-rewrite field is set to no by default. For applications with high latency requirements, this field can be set to yes, otherwise it can be set to no, which is a more secure option for persistence features. Set to yes means that the new write operation is not fsync during rewrite and is temporarily stored in memory until the rewrite is completed. Default is no. Yes is recommended. The default fsync policy for Linux is 30 seconds. Data may be lost for 30 seconds. The default is no.

⑤, auto-aof-rewrite-percentage: the default value is 100. Aof automatically rewrites the configuration, when the current aof file size exceeds how many percent of the last rewritten aof file size, that is, when the aof file grows to a certain size, Redis can call bgrewriteaof to rewrite the log file. When the current AOF file size is twice the size of the AOF file obtained from the last log rewrite (set to 100), a new log rewrite process is automatically started.

⑥ 、 auto-aof-rewrite-min-size:64mb . Set the minimum aof file size that is allowed to be overridden to avoid rewriting when the agreed percentage is reached but the size is still very small.

The ⑦ and aof-load-truncated:aof files may be incomplete at the end. When redis starts, the data of the aof file is loaded into memory. Restart may occur after the operating system of the host where the redis is located goes down, especially if the ext4 file system does not add the data=ordered option. If the redis goes down or terminates abnormally, it will not cause tail incompleteness. You can choose to let redis exit or import as much data as possible. If yes is selected, when the truncated aof file is imported, a log is automatically published to the client and then load. In the case of no, the user must manually redis-check-aof to repair the AOF file. The default is yes.

3. Enable AOF

Just change the appendonly configuration of redis.conf to yes.

The location where AOF saves the file is the same as where RDB saves the file, through the dir configuration of the redis.conf configuration file:

You can get the saved path through the config get dir command.

4. AOF file recovery

After restarting Redis, the AOF file will be loaded.

Exception repair command: redis-check-aof-- fix to repair

5. AOF rewriting

Because AOF persistence means that Redis keeps recording write commands to AOF files, with the progress of Redis, the AOF file will become larger and larger, the larger the file, the larger the server memory and the longer the AOF recovery time. In order to solve this problem, Redis has added a rewriting mechanism. When the size of the AOF file exceeds the set threshold, Redis will start the content compression of the AOF file, keeping only the minimum instruction set that can recover the data. You can use the command bgrewriteaof to re-create.

For example, for the following command:

If you do not rewrite the AOF file, the AOF file will save four SADD commands, and if you use the AOF rewrite, only the following command will remain in the AOF file:

one

Sadd animals "dog"tiger"panda"lion"cat"

In other words, AOF file rewriting is not to reorganize the original file, but to directly read the existing key-value pair on the server, and then replace the previous multiple commands recording this key-value pair with a single command to generate a new file and replace the original AOF file.

AOF file rewrite trigger mechanism: through the auto-aof-rewrite-percentage in the redis.conf configuration file: the default value is 100m, and the auto-aof-rewrite-min-size:64mb configuration, that is, the default Redis records the AOF size when it was last rewritten. The default configuration is triggered when the AOF file size is twice the size of the last rewrite and the file is larger than 64m.

Again, we know that Redis works in a single thread, and if it takes a long time to rewrite AOF, Redis will not be able to process other commands for a long time during AOF rewriting, which is obviously intolerable. Redis to overcome this problem, the solution is to put the AOF rewriting program into a subroutine, which has two benefits:

During the AOF rewrite of ① and child processes, the server process (parent process) can continue to process other commands.

②, the child process has a copy of the data of the parent process, and using the child process instead of the thread can ensure the security of the data without using locks.

The use of child processes solves the above problem, but a new problem also arises: because the child process is still dealing with other commands during AOF rewriting, the new command may also modify the database, making the current database state inconsistent with the rewritten AOF file state.

To solve this problem of inconsistent data state, the Redis server sets up an AOF rewrite buffer, which is used after the child process is created, and when the Redis server executes a write command, it will also send the write command to the AOF rewrite buffer. When the child process completes the AOF rewrite, it sends a signal to the parent process, and after the parent process receives this signal, it calls a function to write the contents of the AOF rewrite buffer to the new AOF file.

This minimizes the impact of AOF rewriting on the server.

6. Advantages and disadvantages of AOF.

Advantages:

① and AOF persistence methods provide a variety of synchronization frequencies. Even if you use the default synchronization frequency to synchronize once per second, Redis will only lose data for 1 second at most.

② and AOF files are constructed using the appended form of the Redis command, so even if Redis can only write fragments of the command to the AOF file, it is easy to modify the AOF file using the redis-check-aof tool.

The format of ③ and AOF files is readable, which also provides users with a more flexible way to deal with. For example, if we accidentally misuse the FLUSHALL command, we can manually remove the last FLUSHALL command before the rewrite, and then use AOF to recover the data.

Disadvantages:

①, Redis,AOF files with the same data are usually larger than RDF files.

②, although AOF provides a variety of synchronization frequencies, by default, the frequency of synchronization once per second also has high performance. However, when the load of Redis is high, RDB has better performance guarantee than AOF.

③ and RDB persist the entire Redis data in the form of snapshots, while AOF simply appends commands executed each time to the AOF file, so RDB is theoretically more robust than AOF. Official documentation also points out that there are some BUG in AOF, and these BUG do not exist in RDB.

So how should we choose between AOF and RDB?

If you can tolerate the loss of data in a short period of time, there is no doubt that using RDB is the best, regular generation of RDB snapshots (snapshot) is very convenient for database backup, and RDB recovery of the dataset is faster than AOF recovery, and the use of RDB can also avoid AOF some hidden bug; otherwise use AOF rewrite. However, in general, it is recommended not to use one persistence mechanism alone, but to use both. In this case, when redis is restarted, the AOF file will be loaded first to recover the original data, because in general, the dataset saved by the AOF file is more complete than the dataset saved by the RDB file. In the later stage of Redis, officials may have integrated the two persistence methods into one persistence model.

The above is all the content of the article "sample Analysis of AOF persistence in Redis". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.