Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to solve the problem of Redis data loss. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

What is an AOF log?

The AOF (Append Only File) log is called "post-write log", that is, the command is executed and the data is written to memory before the log is recorded.

The AOF log (in text form) writes each command received and successfully executed to the text in a certain format (as an append).

"what are the benefits of writing a post-log? "as follows:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

The pre-write log will be recorded regardless of whether the command is executed successfully or not, but the post-write log of Redis will be written only if the command is executed successfully, which avoids the existence of incorrect commands in the log.

At the same time, because the log is written only after the command is executed successfully, it does not block the execution of the current command.

But AOF logs also have "potential risks", which are analyzed as follows:

1. Because it is a post-write log, if the server suddenly goes down after the successful execution of the command and before the log is written to disk, when the data is restarted and recovered, this part of the data must not exist in the log file and will be lost. (in the case that it cannot be recovered from the background database)

two。 Although it does not block the execution of the current command, because logging is also in the main thread (Redis is single-threaded), if the log is suddenly blocked while writing to disk, it will certainly affect the execution of the next command.

To address the above risks, the AOF log provides three write-back strategies.

Three writeback strategies

The AOF mechanism provides three write-back policies, all of which are configured in appendfsync, as follows:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Always (synchronous writeback): write the log to disk synchronously immediately after the command is executed.

Everysec (write back per second): after the command is executed, the log is written to the memory buffer of the AOF file, and the contents of the buffer are written to disk every other second.

No (operating system controlled writeback): after each write command is executed, the log is first written to the memory buffer of the AOF file, and it is up to the operating system to decide when to write the contents of the buffer back to disk.

In fact, none of the three writeback strategies can solve the problems of blocking and data loss of the main thread. The analysis is as follows:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Synchronous writeback: basically no data loss, but each step will have a slow disk operation, which will inevitably affect the performance of the main thread.

Write back per second: write to the AOF log file once a second, but you will still lose one second of data in case of downtime.

Operating system-controlled writeback: after the buffer is written, it is written to disk, but the data is always lost if it is down within the buffer time.

The advantages and disadvantages of the above three strategies are summarized as follows:

What if the log file is too large?

As the amount of data increases, AOF log files will inevitably be very large, which will cause both write and recovery of data to become very slow. At this point, AOF provides a "rewriting mechanism" to solve this problem.

The ❝rewriting mechanism is simple to understand, that is, Redis creates a new AOF log file and writes the final value of each key-value pair to the log file with a command.

For example, if the key-value pair key1:value1 is read, the rewriting mechanism will record the following command in the new AOF log file:

Set key1 value1

In fact, the final value of recording multiple modifications is recorded in a new AOF log file, so that the command can be executed directly when the data is recovered.

"Why can the rewriting mechanism shrink the file? "when a key value is modified many times, commands that modify the key value several times will be recorded in the AOF log file. The rewriting mechanism generates a" write "command for it based on the latest state of the key value, so that the" multiple "command in the old file becomes a" one "command in the new log after rewriting.

The author draws a rewriting flow chart for reference only, as follows:

Rewrite mechanism process

Will AOF rewriting block the main thread?

Although AOF rewriting can reduce the size of log files and reduce the time of logging and data recovery, it is a very time-consuming process to write the rewritten logs of the entire database to disk when the amount of data is very large. Won't it block the main thread?

"the answer is: do not block the main thread"; because the AOF rewriting process is done by the backstage child process bgrewriteaof, which is also to avoid blocking the main thread, resulting in database performance degradation.

In fact, the process of rewriting is divided into two stages: "one copy, two logs".

"one copy": each time a rewrite is performed, the main thread fork a child thread bgrewriteaof, and the main thread copies the memory data to the child thread, which contains the latest data of the database. The child thread can then rewrite the AOF without affecting the main thread.

What is "two logs"? As follows:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

The first log: the child thread rewrite does not block the main thread, while the main thread still processes the request, and the AOF log is still being recorded, so that even if there is an outage, the data is complete. The first log is the log that the main thread is using.

The second log: refers to the new AOF rewrite log; operations during the rewrite process are also written to the rewrite log buffer so that the rewrite log does not lose the latest operation. When all the operation records of the copied data are rewritten, the latest operations of rewriting the log records are also written to the new AOF file to ensure the record of the latest state of the database. At this point, we can replace the old file with the new AOF file.

❝"summary": when Redis performs an AOF rewrite, it will fork a child thread (without blocking the main thread) and make a memory copy for the rewrite, and then use two logs to ensure that the newly written data will not be lost during the rewrite.

Shortcomings of AOF

Although the AOF log files will be greatly reduced after log rewriting, it is still a command (because of single thread, can only be executed sequentially) in the process of data recovery, and the recovery process is very slow.

Summary

AOF, which records operation commands one by one, provides three write-back strategies to ensure the reliability of data, namely, Always, Everysec and No. These three strategies are from high to low in reliability and from low to high in performance.

In order to prevent the log file from being too large, Redis provides a mechanism for rewriting, each rewrite fork a child thread, copy memory data for rewriting, reduce multiple commands to a command that generates key-value pairs, and finally rewrite the log as a new log.

What is RDB?

RDB (Redis DataBase) is another way of persistence: memory snapshots.

❝RDB records in-memory data "at some point", not an operation command.

This method is similar to taking pictures, keeping only the image of a certain moment. A memory snapshot is to write the state of a moment to disk in the form of a file. In this way, even if the data is down, the data will not be lost, and this snapshot file is called the RDB file.

Because ❝records memory data at a certain time, the data recovery is very fast, and there is no need to execute the recorded commands one by one like the AOF log.

What data is snapped?

In order to ensure the reliability of the data, Redis performs a "full snapshot", that is, writing all the data in memory to disk.

With the increase of the amount of data, writing all the data to disk at once is bound to cause thread blocking, which is related to the performance of Redis.

Redis provides two commands for thread blocking, as follows:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Save: executes in the main thread, causing the main thread to block.

Bgsave:fork is a child process designed to write to RDB files, avoiding blocking from the main thread, which is the default configuration of Redis.

In this way, you can use the bgsave command to perform a full snapshot, which not only ensures the reliability of the data but also avoids the blocking of the main thread.

Can the data be modified during the snapshot?

While the child thread performs the full snapshot, the main thread is still accepting the request, and there must be no problem in reading the data, but if the data is modified, how can the integrity of the snapshot be guaranteed?

"take a chestnut": I take a full snapshot at T time, assuming that the amount of data is 8 gigabytes, and the process of writing to disk takes at least 20s. During this 20s time, once the data in memory is modified, the integrity of the snapshot is destroyed.

But if the data cannot be modified during the snapshot, it will have a huge impact on the performance of Redis. How does Redis solve this problem?

❝Redis uses the write-time replication technology (Copy-On-Write, COW) provided by the operating system to handle write operations normally while performing snapshots.

In fact, the bgsave command will fork a child thread that shares all the data in memory, and the child thread will read the data in the main thread's memory and write them to the RDB file.

Copy while writing to ensure that the data can be modified

As shown in the figure above, the reading of A for the key value does not affect the child thread, but if the main thread modifies a piece of data in memory (such as key value pair D), the data will be copied and then the bgsave child thread will write it to the RDB file.

How often do you take a snapshot?

A snapshot only records data at a certain time. Once the time is isolated for a long time, once the server goes down, the data for that period of time will be lost.

For example, if a snapshot is taken at T1 time and another snapshot is taken at T1 time, if the server suddenly goes down during this time period, only the snapshot at T1 time is saved, and the data modification during t time period is not recorded (lost). As shown below:

T is down at any time and the snapshot is not executed.

It is obvious from the above picture that "RDB is not a perfect logging scheme". Only by gradually shrinking the t-time can the lost data be shrunk.

"so the question is, can the time be shortened by one second? "that is, a snapshot is performed every second.

❝full snapshot is to record "all" memory data at a certain time, and the execution once per second has a great impact on Redis performance, so the "incremental snapshot" comes out.

Incremental snapshot

Incremental snapshots mean that after a full snapshot is taken, subsequent snapshots only record the modified data, which avoids the overhead of taking full snapshots each time.

The premise of incremental snapshots is that Redis can remember the modified data. In fact, this function is expensive and requires the preservation of complete key-value pairs, which consumes a lot of memory.

❝to solve this problem, Redis uses a mixture of AOF and RDB.

Mixed use of AOF and RDB

This concept was put forward by Redis4.0, which simply means that "memory snapshots are performed at a certain frequency, such as once an hour, between two snapshots, using AOF logs to record all command operations during this period." "

The mixed use of ❝makes it unnecessary to execute memory snapshots frequently, and AOF records not all the operation commands, but the operation commands between two snapshots, so that the AOF log file is not too large and the overhead of AOF rewriting is avoided.

This scheme can not only use the benefits of fast recovery of RDB, but also enjoy the simple advantage of recording only operation commands, which is strongly recommended.

Summary

The RDB memory snapshot records the memory data at a certain time, so it can be recovered quickly; the mixed use of AOF and RDB can make the data recover quickly after downtime, while avoiding the excessive size of AOF log files.

Summary

This paper introduces two schemes of data recovery and persistence, which are AOF and RDB.

What did AOF introduce? As follows:

1. AOF is a post-write log that persists data by recording operation commands.

two。 Because AOF logs after the command is executed, if the server goes down before writing to the disk, the data will be lost; if it is suddenly blocked when writing to the disk, the main thread will be blocked; to solve the above problems, the AOF mechanism provides three write-back strategies, each of which has different advantages and disadvantages.

3. What if the AOF log file is too large? AOF rewrites a new log file by fork a child thread (sharing the memory of the main thread and recording the write command of the latest data), while the child thread rewrites to avoid blocking the main thread.

What did RDB introduce? As follows:

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

A RDB is a memory snapshot that records memory data at a given time, not an operation command.

Redis provides two commands, save and bgsave, to execute the full snapshot. The difference between these two commands is that save is executed in the main thread, which is bound to block the main thread, and bgsave is a child thread in fork, sharing memory.

Through the write-time replication technology of the operating system, RDB can guarantee that the main thread can modify the snapshot while executing the snapshot.

Because there is an interval between the two snapshots, once the server goes down, the data at the interval is lost, and Redis4.0 starts using AOF logging to record commands executed between snapshots (a mix of AOF and RDB).

On how to solve the loss of Redis data to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.