Shulou(Shulou.com)06/01 Report--

Configuration of AOF persistence and AOF rewriting:

The default AOF mode is disabled, as shown below:

If you want to open it, it is to rewrite no into yes. As shown below:

The default file name is appendonly.aof, and you can also change the file name. The default save directory is also set in the dir configuration item in the configuration file, which shares a directory with RDB. As shown below:

The default synchronization policy is per second, as shown in the following figure:

Let's do something about the database and take a look at the contents of the appendonly.aof file.

It records all write operations.

* 2 indicates 2 parameters $6 indicates that the first parameter length is 6SELECT, the first parameter $1, the second parameter length is 10, and the second parameter is 10

AOF rewriting strategy

The principle of AOF persistence:

When AOF persistence is turned on, when an update operation is made to the database, the update command is appended to the end of the aof_buf buffer and then written to the AOF file by the buffer.

What is recorded in the AOF file is the instruction for the data update operation. The file itself is recorded in text, as shown in the following figure:

When the data needs to be recovered, it can be done by executing the update instructions recorded in the AOF file. Artificially look at the instructions inside, and then manually tap the command can also be completed.

The principle of AOF rewriting implementation:

Because AOF persistence saves database state by recording commands, AOF files will certainly grow over time, which will affect AOF persistence performance and data recovery if not controlled. The following examples illustrate the need for rewriting more vividly:

Let's take a compressed list as an example.

According to the principle of AOF, then the five commands in the red box above should be appended to the AOF file. In fact, we can see that the final state of list is the BCDEF value. In other words, as an example to achieve the final state, you need to add 5 commands. So in the business of reading and writing in a large amount of memory, AOF files grow rapidly. As an example to solve this problem, Redis provides AOF rewriting function.

AOF rewriting is to create a new AOF file to replace the existing AOF file. In fact, AOF rewriting does not operate on the existing old AOF file.

For example, when rewriting, get the latest status of list directly from the database, and then write a rpushlist B C D E F command directly in the new AOF file, thus avoiding the operation of writing 5 items, so that the growth rate of AOF files will slow down and the capacity will not be particularly large.

AOF rewrite program aof_rewrite function to complete the task of creating a new AOF file, but the function will not be called directly by the Redis main process, but will be executed in the background of the child process (BGREWRITEAOF, this command is actually the execution of aof_rewrite, only by the child process to call), then the main process will not be blocked, then the parent process can continue to provide response during the process of performing the rewrite. The whole process is as follows:

When the rewrite is triggered, the parent process calls a function that creates a child process to execute BGREWRITEAOF, which creates a temporary file, and then the parent process continues to provide read and write services to the outside world.

The child process traverses the database and outputs the latest status of each key to a temporary file. During the BGREWRITEAOF process, the parent process writes all update commands to the database to both the AOF buffer and the AOF rewrite buffer (aof_rewrite_buf_blocks), and the AOF buffer (aof_buf) continues to be synchronized to the existing AOF file. (in general, it is not recommended to synchronize the contents of the AOF buffer to the existing AOF file during AOF rewrite This degrades performance, which defaults to NO)

After the AOF rewrite is completed, the child process notifies the parent process, and the parent process calls the signal processing function.

The signal processing function blocks the parent process from providing read and write operations (for a short time, if there is no blocking, there will be data inconsistencies), then write the contents of the AOF rewrite buffer to a new AOF file, and finally replace the existing AOF file with the new AOF file (rename operation)

The APPENDFSYNC option description:

The parameter indicates that always writes and synchronizes everything in the aof_buf buffer to the AOF file, and immediately executes the write () and fsync () system calls. The data is the most secure, but the execution is the slowest, and if there is a failure, only the contents of an event loop will be lost. Everysec writes all the contents of the aof_buf buffer to the AOF file. If the time of the last synchronization of the AOF is more than 1 second, the synchronization is performed, and the write () and fsync () system calls are executed every other second. The data security is in the middle, the execution is fast, and only 1 second of data will be lost. No writes all the contents of the aof_buf buffer to the AOF file, but when synchronization is determined by the operating system, only the write () system call is performed. The write action is efficient, but synchronization is not performed, but a single synchronization takes the longest time, has the lowest data security, and loses all data since the last synchronization.

Here to specifically explain the Linux system file writing and synchronization principle, why say this, because without explaining this process, you will be very difficult to understand the APPENDFSYNC option of the no parameter, if you understand Always as always, always or real-time; and understand everysec as per second, then the meaning of no is not to perform AOF file synchronization? If you don't synchronize files, why turn on AOF persistence?

When Redis calls the appendfsync function, it actually calls a write () function followed by a sync () or fsync () function (the process is the same for any program as long as it wants to write data to disk, with some exceptions).

User space: the area where the regular process is located, initiated by the user, and the code in this area cannot directly access the hardware

Kernel space: the area where the operating system is located, which can communicate with the device controller

When the write () function is called, once the function returns the normal value, we may think that the data has been written to disk, but in fact, when the operating system implements the IO of disk files, in order to ensure the efficiency of IO, it will use a special address space in memory, which is called kernel space. On the other hand, there will be a data buffer in the kernel space that is used as the IO (this buffer is not the same concept as the aof_buf buffer mentioned earlier, although both are in memory), and the function write () is to write the data to the IO buffer in the kernel space.

The IO buffer in kernel space also has a certain size. When the buffer is not full or does not reach a synchronization cycle, the data passed by the write () function will be continuously written to the buffer, and when the buffer is full or reaches a synchronization cycle, the contents of the buffer will be submitted to the output queue, and when the data needs to reach the head of the queue, the real disk IO operation will be performed. Write the data to disk (although this is used to write to disk, the real action is not to move but to copy, and the IO buffer in kernel space will free up the space occupied by the data only after the copy is completed.) This method is called deferred writing.

So there will be a problem, when the call to the write () function does not mean that the data is really saved to disk, but there will be another illusion, that is, when you request the file again, you can display your last updated content, in fact, this content is not read from the disk, but from the user space buffer. Then the problem just mentioned is that if the data is in the IO buffer of the kernel space, when the operating system malfunctions, power outages and other abnormal conditions will cause data loss.

In order to solve the problem of data loss, the Unix system provides three functions: sync, fsync and fdatasync.

Function function sync function returns 0 to indicate success. This function is responsible for pushing the modified contents of the IO buffer in all kernel space to the input queue and then returning. It does not wait for all disk IO operations to be completed. So even if you call the sync function, it doesn't mean you saved it to disk successfully. The fsync function returns 0 to indicate success, unlike sync, which only takes effect on a single file with a specified file descriptor, forces all modified data connected to that file to be transferred to disk, and waits for disk IO to finish, and then returns. When the function returns 0, it really means that it was successfully saved to disk. The database calls fsync () after calling write (). Fdatasync is similar to fsync in that it only affects the data portion of the file and does not involve data attributes, such as inode information. So it requires less write disk operations than fsync.

So if you look at the above, you will know the meaning of the no parameter in APPENDFSYNC.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.