Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to use snapshot and AOF to persist Redis data to the hard disk". In daily operation, I believe many people have doubts about how to use snapshot and AOF to persist Redis data to the hard disk. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer "how to use snapshot and AOF to persist Redis data to the hard disk". Next, please follow the editor to study!

Preface

We know that Redis is a memory server, even if we trust our server enough that there will not be any software or hardware failures, but there may be a sudden power outage and other conditions, resulting in data invalidation in the Redis server. Therefore, we need to back up the data like the traditional relational database and persist the data of Redis in memory to non-volatile media such as hard disk to ensure the reliability of the data.

One of the advantages of persisting data from Redis memory servers to media such as hard drives is that our servers can reuse previous data after restarting, or back up the data to a remote location to prevent system failure.

There are also some scenarios, such as:

For some data that needs a lot of calculation, placed in the Redis server, it is necessary for us to persist the data. If we need to recover the data, we do not need to recalculate it. We just need to simply copy the data from this machine to another Redis server that needs to be restored.

Redis provides us with two different persistence methods: snapshot (Snapshotting) and append-only files (append-only-file).

(1) introduction to nouns

Snapshot (RDB): it is what we commonly call backup, which can back up the data regularly and persist the data in the Redis server to the hard disk.

Only append files (AOF): he will copy the write command to his hard disk when he executes the write command, and when he recovers later, he only needs to re-execute the write command. Similar to our MySQL database, when performing master-slave replication, we use binlog binaries and execute a write command.

(2) General configuration for snapshot persistence:

Save 60 1000 # 60 seconds when there are 1000 writes to perform snapshot creation stop-writes-on-bgsave-error no # whether to continue to execute the write command rdbcompression yes # whether to compress the snapshot file dbfilename dump.rdb # how to name the snapshot file on the hard disk dir. / # location where the snapshot is saved

(3) AOP persistence configuration:

Does appendonly no # use AOF persistence appendfsync everysec # how often do you synchronize writes to your hard disk? when no-appendfsync-on-rewrite no # compresses AOF, can you perform synchronous operations auto-aof-rewrite-percentage # how often do you perform AOF compression auto-aof-rewrite-min-size 64mb # how often do you perform AOF compression dir. / # the location saved by AOF

It should be noted that these two persistence methods can be used either individually or at the same time, and which one needs to be chosen according to the specific situation.

Snapshot persistence

Snapshots are what we call backups. The user can back up the data in Redis memory at a certain point in time, and after creating the snapshot, the user can back up the snapshot. In general, in order to prevent the loss of all data caused by the failure of a single server, we can also copy the snapshot to other servers and create a copy of the data with the same data. In this way, when the data is restored or the server is restarted, the snapshot information can be used for data recovery, and it can also prevent data loss when a single server fails.

However, what we need to pay attention to is that the way we create snapshots does not completely guarantee that our data will not be lost, which we can well understand, because the creation of snapshots is timed. Not every update operation creates a snapshot. When the system crashes, the user loses all data that has changed since the last snapshot was taken. Therefore, the snapshot persistence method is only suitable for scenarios where data is not often modified or some data is lost.

1. How to create a snapshot:

(1) the client creates a snapshot by sending a BGSAVE command to Redis.

When using BGSAVE, Redis calls fork to create a child process, which is then responsible for writing the snapshot to the hard disk, while the parent process continues to process command requests.

Use the scene:

If the user uses the save setting, for example: save 60 1000, the Redis automatically triggers the BGSAVE command when the condition "1000 writes within 60 seconds" is met, starting from the last time Redis created a snapshot.

If the user uses multiple save settings, Redis triggers a BGSAVE command when any save configuration meets the criteria.

(2) the client creates a snapshot by sending a SAVE command to Redis.

The Redis server that receives the SAVE command will no longer respond to requests for any other commands until the snapshot has been created. The SAVE command is not commonly used. We usually use the SAVE command only when we do not have enough memory to execute the BGSAVE command, or even if we wait for the persistence operation to finish.

Use the scene:

When Redis receives a request to shut down the server through the SHUTDOWN command, or receives a standard TERM signal, it executes a SAVE command, blocks all clients, no longer executes any commands sent by the client, and shuts down the server after executing the SAVE command.

2. Notes on using snapshot persistence:

When we use snapshots to save data, if the amount of data in the Redis server is small, for example, when there are only a few GB. Redis creates child processes and saves the data to the hard disk, which takes less time to generate a snapshot than it takes to read the data.

However, as the data grows and Redis takes up more and more memory, BGSAVE will spend more and more time creating child processes. If the Redis server does not have much memory left, this BGSAVE command will cause the system to pause for a long time and may cause the server to become unusable.

By virtual machine category, the time it takes to create child threads:

Therefore, in order to prevent Redis from coming to a standstill when creating child processes, we can consider turning off autosave and manually sending BGSAVE or SAVE for persistence.

Sending BGSAVE manually will also cause a pause, but we can control the time when the command is sent to ensure that the pause does not affect the specific business request.

In addition, it is worth noting that when using the SAVE command, although Redis blocks until the snapshot is generated, it does not need to create a child process, so it does not cause Redis to quiesce as a result of creating a child process, as in BGSAVE. Because of this, SAVE creates snapshots faster than BGSAVE does.

When creating a snapshot, we can execute it regularly when there are few business requests, such as 3: 00 or 4: 00 in the morning.

AOF persistence

AOF persistence records changes in the data by writing the executed write command to the end of the AOF file. In this way, when we recover the data, we only need to execute the AOF file from beginning to end to recover the data.

First, open the AOF persistence option

We can open AOF by using the following command:

Appendonly yes

We configure the synchronization frequency of the AOF file with the following command:

Appendfsync everysec/always/no

Second, the difference of appendfsync synchronization frequency.

The difference of appendfsync synchronization frequency is shown below:

(1) always can save no data very well, but this synchronization strategy requires a lot of write operations on the hard disk, so the speed of Redis processing commands will be limited by the performance of the hard disk.

An ordinary hard disk can only handle about 200write commands per second, while using solid-state SSD can handle tens of thousands of write commands per second, but write only one command at a time, this can only be blamed for constantly writing a very small amount of data, which may lead to serious write magnification problems, and this decline seriously affects the service life of solid-state drives.

(2) in the way of everysec, Redis synchronizes AOF files with a frequency of once per second. In this way, both data security and write performance can be considered.

Redis has the same performance of synchronizing AOF files per second as it does without any persistence features, and updating once per second ensures that lost data is generated within a second in the event of a failure.

(3) in the way of no, Redis will not perform any displayed synchronization operation on the AOF file, but it is up to the operating system to decide when to synchronize the AOF file.

This command generally does not have much impact on the performance of Redis, but Redis servers that use this option lose a variable amount of data when the system fails.

In addition, when the user's hard disk is not fast enough to process writes, then the buffer is filled with data waiting to be written to the hard disk, Redis's write operations will be blocked and Redis will slow down the processing of command requests. For this reason, this option is generally not recommended.

3. Rewrite / compress AOF files

With the increase of the amount of data, the files of AOF may be very large, so it will take a long time for each data recovery. In order to solve the growing number of AOF files, users can send BGREWRITEAOF commands to Redis. This command will rewrite AOF files by removing redundant commands from AOF files, so that the physical examination of AOF files becomes as small as possible.

How BGREWRITEAOF works is similar to how BGSAVE works: Redis creates a child process, which is then responsible for rewriting the AOF file.

Because the child process is created when the AOF file is rewritten, the performance and memory footprint problems caused by snapshot persistence due to the creation of the child process will also occur when the AOF file is rewritten.

Setting conditions for triggering rewriting / compressing AOF files

AOF automatically executes BGREWRITEAOF by setting the auto-aof-rewrite-percentage and auto-aof-rewrite-min-size options.

The specific meaning, as can be seen from an example, is as follows:

Auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb

Indicates that when the file size of the current AOF is larger than 64MB, and the AOF file is at least twice as large as it was after the last override, Redis executes the rewrite BGREWRITEAOF command.

If AOF rewrites are performed too frequently, you can set the value of the auto-aof-rewrite-percentage option to more than 100. this most occasional occurrence allows Redis to perform a rewrite operation after the size of the AOF file becomes larger, but it also takes longer to perform during data recovery.

Verify snapshot files and AOF files

No matter which method is used for persistence, Redis provides two command-line programs when we recover the data:

Redis-check-aofredis-check-dump

They can check the status of snapshots and AOF files in the event of a system failure, and repair the files if necessary.

If the user specifies the-- fix parameter when running the redis-check-aof command, the program will repair the AOF file.

The way the program fixes the AOF file is simple: it scans the given AOF file for incorrect or incomplete commands, and when it finds the first error command, the program deletes the error command and all commands after the error command, leaving only those correct commands that precede the error command. In most cases, it is the incomplete write command at the end of the AOF file that is deleted.

At this point, the study on "how to use snapshots and AOF to persist Redis data to the hard disk" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.