Shulou(Shulou.com)05/31 Report--

This article shares with you the content of a sample analysis of redis persistence. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Why do you need persistence?

Because Redis is a kind of memory database, that is, when the server is running, the system allocates part of the memory to store the data. Once the server goes down or suddenly goes down, the data in the database will be lost. In order to save the data even if the server shuts down suddenly, the data must be saved from memory to disk by persistence.

For a persistent program, the process for writing data from the program to the computer's disk is as follows:

1. The client sends a write instruction to the database (when the data is in the memory of the client)

2. The database receives the write instruction and data (the data is in the memory of the server at this time)

3. The database initiates a system call to write the data to disk (when the data is in the memory of the kernel)

4. The operating system transfers the data to the disk controller (the data is now in the disk cache)

5. The disk controller performs the operation of actually writing data to the physical medium (such as a disk)

If you only consider the database level, the data will be secure after the third stage, at this time, the system call has been initiated, even if the database process crashes, the system call will continue and the data can be written to disk smoothly. After this step, in step 4, the kernel will save the data from the kernel cache to the disk cache, but for the sake of system efficiency, this action will not be performed too often by default, probably once in 30s, which means that if this step fails or the server suddenly shuts down while doing this step, then 30s of data may be lost. This more common catastrophic problem also needs to be considered.

POSIX API also provides a system call for the kernel to force cached data to be written to disk, most commonly fsync system calls.

Int fsync (int fd)

The fsync function works on only one file specified by the file descriptor fd and waits for the write disk operation to finish before returning. Each time fsync is called, a write operation is initialized and the data from the buffer is written to disk. The fsync () function blocks the process when it completes the write operation, and if other threads are writing the same file, it also blocks other threads until the write operation is complete.

Persistence

Persistence is a mechanism for converting program data between persistent state and instantaneous state. For the program, the data is in memory while the program is running, and if it is not synchronously written to the disk in time, then once the power is cut off or the program suddenly breaks down, the data will be lost. Only by synchronizing the data to the disk in time can the data be saved permanently, not because of the validity of the downtime image data. Persistence is the process of synchronizing data from the program to the disk.

Persistence of Redis

Redis has two persistence methods: RDB and AOF. RDB is the way of snapshot files. Redis saves the current data of redis to a * .rdb file by executing SAVE/BGSAVE commands to perform data backup, which saves all data sets. AOF means that the server appends the command of redis write operation to the * .aof file at a specified time by reading the configuration. It is an incremental persistence method.

RDB

The RDB file is implemented through the SAVE or BGSAVE command. The SAVE command blocks the Redis service process until the RDB file is created. The BGSAVE command creates the RDB file through the fork child process, the parent process and the child process share the data segment, the parent process continues to provide read and write services, and the child process realizes the backup function. The BGSAVE phase is copied only when the shared segment needs to be modified, that is, COW (Copy On Write). SAVE can create RDB files by setting multiple save conditions, and SAVE operations can be performed in the background as long as one of the conditions is met.

The implementation code for the SAVE and BGSAVE commands is as follows:

Void saveCommand (client * c) {/ / BGSAVE cannot execute SAVEif (server.rdb_child_pid! =-1) {addReplyError (c, "Background save already in progress"); return;} rdbSaveInfo rsi, * rsiptr;rsiptr = rdbPopulateSaveInfo (& rsi); / / call the rdbSave function to perform a backup (blocking the current client) if (rdbSave (server.rdb_filename,rsiptr) = = C_OK) {addReply (cdepartment shared.ok);} else {addReply (cfield shared.err) }} / * * BGSAVE command to implement [optional parameter "schedule"] * / void bgsaveCommand (client * c) {int schedule = 0 / * the effect of SCHEDULE parameter modification of BGSAVE when AOF is being executed * BGSAVE will be executed later instead of reporting an error * it can be understood as: BGSAVE is put on the agenda * / if (c-> argc > 1) {/ / parameter can only be "schedule" if (c-> argc = = 2 & &! strcasecmp (c-> argv [1]-> ptr, "schedule")) {schedule = 1;} else {addReply (cForce shared.syntaxerr); return }} / / BGSAVE is executing, not operating if (server.rdb_child_pid! =-1) {addReplyError (c, "Background save already in progress");} else if (server.aof_child_pid! =-1) {/ / aof is being executed, if schedule==1,BGSAVE is put on the agenda if (schedule) {server.rdb_bgsave_scheduled = 1 / addReplyStatus (c, "Background saving scheduled");} else {addReplyError (c, "An AOF log rewriting in progress: can't BGSAVE right now. "" Use BGSAVE SCHEDULE in order to schedule a BGSAVE whenever "" possible. ");}} else if (rdbSaveBackground (server.rdb_filename,NULL) = = C_OK) {/ / otherwise call rdbSaveBackground to perform backup operation addReplyStatus (c," Background saving started ");} else {addReply (cmam shared.err);}}

After you have the RDB file, if the server shuts down, or if you need to add another server, after restarting the database server, you can restore the previously backed up data by loading the RDB file. However, bgsave will take a long time, not real-time, and will lead to the loss of a large amount of data during downtime.

AOF (Append Only File)

The RDB file saves the key-value pairs of the database, while AOF saves the write commands executed by the database.

The implementation process of AOF has three steps:

Append- > write- > fsync

Append appends commands to the AOF buffer, write writes the contents of the buffer to the program buffer, and fsync writes the contents of the program buffer to the file. When AOF persistence is enabled, every time the server executes a command, it will append the command to the aof_buf buffer of the redisServer structure in protocol format. The specific protocol is not described here.

There is a configuration option for the duration of AOF persistence: appendfsync. This option has three values: always: everything is written and synchronized to the aof file everysec: writes the contents of the aof_buf buffer to the AOF file, if the no of the last synchronized AOF file: writes everything in the aof_buf buffer to the AOF file, but does not synchronize the AOF file, it is up to the operating system to decide when to synchronize, usually 30s by default.

AOF persistence mode every write command will be appended to the AOF file. As the server continues to run, the AOF file will become larger and larger. In order to avoid the file generated by AOF being too large, the server will rewrite the AOF file and merge the same commands that operate on the same key, thus reducing the file size.

For example, to save an employee's name, gender, and other information:

> hset employee_12345 name "hoohack"

> hset employee_12345 good_at "php"

> hset employee_12345 gender "male"

Just enter the state of the hash key, the AOF file needs to save three commands, if there are other, such as delete, or update the value of the operation, then there will be more commands, the file will be larger, after rewriting, you can appropriately reduce the size of the file.

The implementation principle of AOF rewriting is to first the database in the server, and then traverse the database, find out all the key objects in each database, obtain the key and value of the key-value pair, and rewrite the key-value pair according to the type of key. For example, the above example can be merged into the following command:

> hset employee_12345 name "hoohack" good_at "php" gender "male".

AOF rewriting performs a lot of write operations, and Redis is single-threaded, so if a server invokes rewriting directly, the server cannot handle other commands, so the Redis server has a new process to perform AOF rewriting.

Redis performs the process of rewriting:

When the child process performs AOF rewriting, after receiving the command from the client, the server first executes the command sent by the client, then appends the executed write command to the AOF buffer, and at the same time appends the executed write command to the AOF rewrite buffer. When the child process completes the rewrite, it sends a complete signal to the server, and the server appends everything in the AOF rewrite buffer to the AOF file, and then atomically overwrites the existing AOF file.

Advantages and disadvantages of RDB and AOF

RDB persistence mode can only read data from the server to load files in the backup into the program, while AOF mode must create a pseudo client to execute.

The RDB file is small and saves the data before a certain point in time, so it is suitable for disaster recovery and master-slave synchronization.

RDB backup takes a long time, and if the amount of data is large, some data may be lost in the event of downtime. In addition, RDB is implemented only when certain conditions are met through configuration, and this part of the data will also be lost if it goes down during this period of time.

AOF mode, in the case of the same data set, the file size will be larger than the RDB mode.

The persistence mode of AOF is also different through configuration. The default configuration is synchronization per second, the fastest mode is to synchronize every command, and the worst way is to wait for the system to execute fsync to synchronize the buffer to disk files. Most operating systems are 30s. It is usually configured to synchronize once per second, so up to 1 second of data will be lost.

What kind of synchronization is better?

The combination of RDB and AOF. Start a scheduled task to back up one copy of the server's current status data every hour, named after the date and hour, and another scheduled task to regularly delete invalid backup files (such as 48 hours ago). AOF is configured as 1s once. In this way, up to 1 second of data will be lost, and if the redis has an avalanche, it can quickly return to the state of the previous day without stopping the service.

Thank you for reading! This is the end of this article on "sample Analysis of redis persistence". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.