Shulou(Shulou.com)06/01 Report--

Recently, I learned redis after work. Here's a simple understanding of redis persistence.

Persistence mechanism provided by Redis

Redis is a distributed NoSQL database system for "key-value" type data, which has the advantages of high performance, persistent storage and adapting to high concurrency application scenarios. Although it started late, it developed rapidly.

1. How does Redis persistence work?

What is persistence? To put it simply, it is to put the data on a device that will not be lost after a power outage, that is, the hard disk as we usually understand it.

First of all, let's take a look at what the database does when writing. There are five main processes:

(1) the client sends a write operation to the server (the data is in the client's memory).

(2) the database server receives the data of the write request (the data is in the memory of the server).

(3) the server calls the system call write to write the data to disk (the data is in the buffer of the system memory).

(4) the operating system transfers the data in the buffer to the disk controller (the data is in the disk cache).

(5) the disk controller writes the data to the physical media of the disk (the data actually falls on the disk).

1. Fault analysis:

There are roughly five processes above for write operations. Let's take a look at various levels of failures by combining the above five processes:

(1) when the database system fails, the system kernel is still intact. So at this point, as long as we have completed step 3, then the data is secure, because the subsequent operating system will complete the next few steps to ensure that the data will eventually fall on disk.

(2) when the system is powered off, all the caches mentioned in the above five items will be invalidated, and the database and operating system will stop working. Therefore, only when the data is completed in step 5, can we ensure that the data will not be lost after the power outage.

Through the understanding of the above five steps, we may want to figure out some of the following questions:

(1) how often does the database call write to write the data to the kernel buffer?

(2) how long will the kernel write the data in the system buffer to the disk controller?

(3) when does the disk controller write the data in the cache to the physical media?

For the first question, there is usually full control at the database level.

For the second problem, the operating system has its default policy, but we can also force the operating system to write data from the kernel to the disk controller through the fsync series commands provided by POSIX API.

For the third problem, it seems that the database can no longer be reached, but in fact, in most cases, disk caching is set to be turned off or only opened for read caching, that is, writes are not cached and written directly to disk.

The recommended practice is to turn on write caching only if your disk device has a spare battery.

2. Data corruption

The so-called data corruption means that the data cannot be recovered. what we are talking about above is how to ensure that the data is actually written to disk, but writing to disk may not mean that the data will not be corrupted. For example, we may perform two different write operations on one write request, which may cause one write operation to be completed safely when an accident occurs, but the other has not yet taken place. If the data file structure of the database is not organized properly, it may lead to the situation that the data can not be recovered at all.

There are also usually three strategies for organizing data to prevent data files from being corrupted to unrecoverable situations:

(1) the first is the roughest processing, which is not to ensure the recoverability of the data through the organizational form of the data. Instead, it is restored through data backup after the data file is corrupted by configuring data synchronization backup. In fact, MongoDB does not turn on operation logging, as is the case when configuring Replica Sets.

(2) the other is to add an operation log based on the above, and write down the behavior of the operation each time, so that we can recover the data through the operation log. Because the operation log is written sequentially, it does not occur that the operation log cannot be recovered. This is also similar to what happens when operation logging is enabled by MongoDB.

(3) it is safer for the database not to modify the old data, but to complete the write operation in an additional way, so that the data itself is a log, so that the data can never be recovered. In fact, CouchDB is an excellent example of this.

2. Redis provides RDB persistence and AOF persistence

1. The advantages and disadvantages of RDB mechanism.

RDB persistence refers to writing a snapshot of a dataset in memory to disk within a specified time interval.

It is also the default persistence method, which is to write in-memory data to a binary file as a snapshot, and the default file name is dump.rdb.

A, advantage

(1) once you use this method, your entire Redis database will contain only one file, which is very convenient for backup. For example, you may plan to file some data in less than a day.

(2) to facilitate backup, we can easily move a RDB file to other storage media.

(3) the recovery speed of RDB is faster than that of AOF when recovering large datasets.

(4) RDB can maximize the performance of Redis: the only thing the parent process needs to do when saving the RDB file is to fork out a child process, and then the child process will handle all the subsequent save work, and the parent process does not need to perform any disk Icano operations.

B, disadvantage

(1) if you need to avoid losing data in the event of a server failure, RDB is not for you. Although Redis allows you to set different save points (save point) to control how often the RDB file is saved, because the RDB file needs to save the state of the entire dataset, it is not an easy operation. So you may save the RDB file at least once in 5 minutes. In this case, in the event of a downtime, you may lose data for several minutes.

(2) every time RDB is saved, Redis needs to fork () a child process, and the child process carries out the actual persistence work. When the dataset is large, fork () can be very time-consuming, causing the server to stop processing the client in so-and-so milliseconds; if the dataset is very large and CPU time is very tight, the stop time may even be as long as a full second. Although AOF rewriting also requires fork (), there is no loss of data durability no matter how long the execution interval of AOF rewriting is.

You can configure how snapshots are persisted automatically. We can configure redis to take snapshots automatically if more than m key are modified in n seconds. Here is the default snapshot save configuration.

Save 9001 # 900s if more than one key is modified, initiate snapshot save

Save 30010 # 300s if more than 10 key are modified, initiate snapshot save

Save 60 10000

2. RDB file saving process

(1) redis calls fork and now has a child process and a parent process.

(2) the parent process continues to process the client request, and the child process is responsible for writing the memory contents to the temporary file. Because os's write-time copy mechanism (copy on write) parent and child processes share the same physical page, when the parent process processes the write request, os creates a copy of the page to be modified by the parent process instead of writing to the shared page. Therefore, the number in the address space of the child process is a snapshot of the entire database at the time of fork.

(3) when the child process has finished writing the snapshot to the temporary file, replace the original snapshot file with the temporary file, and then the child process exits.

Client can also use the save or bgsave command to tell redis to do a snapshot persistence. The save operation holds the snapshot in the main thread, and because redis uses a main thread to process all client requests, this way blocks all client requests. So it is not recommended.

Another point to note is that each snapshot persistence is a complete write of memory data to disk, not incremental synchronous dirty data. If there is a large amount of data and a lot of write operations, it will inevitably cause a large number of disk io operations, which may seriously affect performance.

3. AOF file saving process

Redis appends every write command received to the file through the write function (default is appendonly.aof).

When redis restarts, it rebuilds the contents of the entire database in memory by re-executing the write command saved in the file. Of course, because os caches changes made by write in the kernel, it may not be written to disk immediately. In this way, it is still possible to lose some of the changes in aof persistence. However, we can tell redis through the configuration file when we want to force os to write to disk through the fsync function. There are three ways to do the following (default: fsync once per second)

Appendonly yes / / enable aof persistence mode

# appendfsync always / / Force to write to disk immediately every time you receive a write command, the slowest, but guaranteed to be fully persistent, not recommended

Appendfsync everysec / / is forced to write to disk once per second, which makes a good compromise between performance and persistence. It is recommended

# appendfsync no / / totally dependent on os, with the best performance and no guarantee of persistence

The AOF approach also brings another problem. Persistent files will get bigger and bigger. For example, if we call the incr test command 100 times, all 100 commands must be saved in the file, but 99 of them are redundant. Because in order to restore the state of the database, it is enough to save a set test 100in the file.

To compress the persistence file of aof. Redis provides the bgrewriteaof command. Upon receipt of this command, redis saves the data in memory to a temporary file in a command similar to a snapshot, and finally replaces the original file. The specific process is as follows

(1) redis calls fork, and now there are two processes: father and son.

(2) the child process writes the command to rebuild the database state to the temporary file according to the database snapshot in memory.

The parent process continues to process client requests, except for writing write commands to the original aof file. At the same time, write commands received are cached. This ensures that there will be no problems if the child process rewrite fails.

When the child process writes the contents of the snapshot to a temporary file in a commanded manner, the child process signals the parent process. The parent process then writes the cached write command to the temporary file.

(3) now the parent process can replace the old aof file with a temporary file and rename it, and the write commands received later begin to append to the new aof file.

(4) it should be noted that the operation of rewriting the aof file does not read the old aof file, but rewrites a new aof file with the entire database contents in memory in a command way, which is similar to the snapshot.

A, advantage

(1) using AOF persistence makes Redis very durable (much more durable): you can set different fsync policies, such as no fsync, fsync per second, or fsync each time a write command is executed. AOF's default policy is to fsync once per second, under this configuration, Redis can still maintain good performance, and even if a failure occurs, it will only lose data for up to one second (fsync executes in the background thread, so the main thread can continue to work hard on command requests).

(2) the AOF file is a log file (append only log) that only performs append operations, so writing to the AOF file does not require seek. Even if the log contains commands that are not fully written for some reason (such as the disk is full at the time of writing, write stops, etc.), the redis-check-aof tool can easily fix this problem.

(3) Redis can automatically rewrite AOF in the background when the size of the AOF file becomes too large: the new AOF file after rewriting contains the minimum set of commands needed to restore the current dataset. The entire rewrite operation is perfectly safe because Redis continues to append commands to existing AOF files during the creation of new AOF files, and existing AOF files will not be lost even if downtime occurs during rewriting. Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and starts appending the new AOF file.

(4) the AOF file saves all the write operations performed to the database in an orderly manner, and these write operations are saved in the format of Redis protocol, so the contents of the AOF file are very easy to read and parse. Exporting (export) the AOF file is also very simple: for example, if you accidentally execute the FLUSHALL command, but as long as the AOF file is not rewritten, simply stop the server, remove the FLUSHALL command at the end of the AOF file, and restart Redis, and you can restore the dataset to the state it was before the FLUSHALL execution.

B, disadvantage

(1) for the same dataset, the volume of the AOF file is usually larger than that of the RDB file.

(2) depending on the fsync strategy used, AOF may be slower than RDB. In general, the performance of fsync per second is still very high, and turning off fsync can make AOF as fast as RDB, even under heavy loads. However, RDB can provide a more guaranteed maximum latency (latency) when dealing with large write loads.

(3) AOF has had such a bug in the past: due to individual commands, when the AOF file is reloaded, the dataset cannot be restored to its original state. (for example, the blocking command BRPOPLPUSH has caused such a bug. Tests are added to the test suite: they automatically generate random, complex data sets and make sure everything is all right by reloading the data Although this kind of bug is not common in AOF files, it is almost impossible for RDB to have this kind of bug by comparison.

3. Choice

In general, if you want to achieve data security comparable to PostgreSQL, you should use both persistence features.

If you are very concerned about your data, but can still withstand data loss within a few minutes, then you can only use RDB persistence.

AOF is my personal preference for the rest.

4. What if the AOF file goes wrong?

The server may stop while the program is writing to the AOF file, and if the downtime causes an error in the AOF file (corrupt), then Redis will refuse to load the AOF file when it is restarted, thus ensuring that the consistency of the data will not be broken.

When this happens, you can repair the faulty AOF file in the following ways:

(1) create a backup of the existing AOF file.

(2) use the redis-check-aof program that comes with Redis to repair the original AOF file.

$redis-check-aof-fix

(3) [optional] use diff-u to compare the backup of the repaired AOF file and the original AOF file to see the difference between the two files.

Restart the Redis server, wait for the server to load the repaired AOF file, and restore the data.

Author: SEian.G (hard practice changes in 72, but it is difficult to laugh at 81)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.