Shulou(Shulou.com)06/01 Report--

I. Overview

The powerful performance of Redis is largely due to the fact that all data is stored in memory, but when Redis is restarted, all data stored in memory will be lost, which in many cases cannot be tolerated. So, we need to persist the data in memory! Typical scenarios where data needs to be persisted are as follows:

Use Redis as a database and Redis as a cache server, but caching miss will have a great impact on performance. When all caches fail at the same time, it will cause a service avalanche and become unresponsive.

This article introduces two data persistence methods supported by Redis.

2. Persistence of Redis data

Redis supports two data persistence methods: RDB and AOF. The former persists the data in memory to the hard disk regularly according to the configured rules, while the latter records the command after each write command. The two persistence methods can be used separately, but they are usually used in combination.

1. RDB mode

Persistence in RDB mode is done through snapshots. When a certain rule is met, a copy of all the data in memory is generated and stored on the hard disk, a process called "snapshot", and Redis takes a snapshot of the data in the following situations:

Take automatic snapshots according to configuration rules; users execute SAVE, BGSAVE commands; execute FLUSHALL commands; when performing replication (replication).

The scene where the snapshot is performed

(1) automatic snapshots based on configuration

Redis allows users to customize snapshot conditions and automatically execute snapshots when the conditions are met. By default, Redis stores data snapshots in a binary file on disk called dump.rdb. In addition, we can also modify the frequency of dump snapshots on the Redis server through the configuration file. After opening the redis.windows.conf file, we search save and see the following configuration information:

Notice the last three lines, which indicate:

After 900s (15 minutes), if at least one key changes, the dump memory snapshot

After 300 seconds (5 minutes), if at least 10 key changes, the dump memory snapshot

After 60 seconds (1 minute), if at least 10000 key changes, the dump memory snapshot.

Each snapshot condition has a single line, and there is an or (| |) relationship between them. As long as any one of them is satisfied, the snapshot will be taken. The first parameter T after configuring save above is time, in seconds, and the second parameter M is the number of keys changed, which means that when the number of keys changed in time T is greater than M, snapshots are taken automatically. For example, save 9001 means that if the number of keys changed is greater than 1 within 15 minutes (900s), the snapshot operation will be performed automatically.

(2) execute SAVE or BGSAVE commands

In addition to letting Redis take snapshots automatically, when we need to restart, migrate, and back up Redis, we can also manually execute SAVE or BGSAVE commands to take the initiative to snapshot operations.

SAVE command: when executing the SAVE command, Redis synchronously performs snapshot operations, during which all requests from the client are blocked, so you should avoid using this command when there is more data in the database.

BGSAVE command: as can be seen from the name of the command, the difference between this command and the SAVE command is that the snapshot operation of this command is carried out asynchronously in the background, and it can handle requests from the client at the same time. After executing the BGSAVE command, Redis will immediately return OK to start the snapshot operation. If you want to know whether the snapshot operation has been completed, you can use the LASTSAVE command to return the time when the last snapshot was successfully executed, and the result is a Unix timestamp.

(3) execute FLUSHALL command

When you execute the FLUSHALL command, Redis clears all data in the database. It should be noted that regardless of whether the condition for automatic snapshot is triggered by the process of emptying the database, Redis will perform a snapshot operation as long as the automatic snapshot condition is not empty, and executing the FLUSHALL command will not perform a snapshot operation when no automatic snapshot condition is defined.

(4) perform replication

When the master-slave mode is set, Redis takes an automatic snapshot at replication initialization.

Snapshot principle

By default, Redis stores snapshot files in the dump.rdb file in the working directory of the current Redis process. You can specify the storage path and file name of the snapshot file through the parameters dir and dbfilename in the configuration file. The default storage path and file name are shown in the following figure:

The snapshot performs the following process:

(1) Redis uses the fork function to make a copy of the current process (parent process) (child process)

(2) the parent process continues to process requests from the client, and the child process begins to write data in memory to temporary files on the hard disk.

(3) after the child process has written all the data, replace the old RDB file with the temporary file, and a snapshot operation is completed.

It is important to note that:

The operating system (Unix-like operating system) will use the copy-on-write policy when executing fork, that is, when the fork function occurs, the parent process and the child process share the same piece of memory data. When the parent process needs to modify a piece of data (such as executing a write command), the operating system will make a copy of the data to ensure that the child process is not affected. So the RDB file stores in-memory data from the moment the fork operation is performed. So in theory, there will be data loss in the RDB way (those modified after fork are not written into the RDB file).

As can be known from the above introduction, the RDB file will not be modified when the snapshot is in progress, and the old RDB file will be replaced with a temporary file only when it is completed, so make sure that the RDB file is complete at any time. This enables us to back up Redis data by backing up RDB files regularly. RDB files are compressed binary files, so the space occupied is less than the size of the data in memory, which is more convenient for transmission.

When Redis starts, it automatically reads the RDB snapshot file and loads the data from the hard disk to memory, and the duration of this process varies according to the quantity. Generally speaking, it takes 20-30 seconds for a snapshot file to record 10 million string type keys and the size of 1GB to be loaded into memory.

Example

Here is a demonstration of RDB persistence. First of all, the snapshot rules are configured as follows:

Save 900 1save 300 10save 60 10000dbfilename dump.rdbdir. /

Start the Redis service:

Then set a key value through the client:

Now force the kill Redis service and execute the shutdown command:

Now go to the D:\ Redis_x64_321\ directory and see the snapshot file dump.rdb of Redis:

Now restart Redis, and then use the client connection to check that the previously set key still exists:

You can see that the previously set key was restored through the snapshot file dump.rdb after the Redis restart.

2. AOF mode

When using Redis to store non-temporary data, you generally need to open AOF persistence to reduce data loss caused by process termination. AOF can append every write command executed by Redis to the hard disk file, this process will obviously reduce the performance of Redis, but in most cases this effect is acceptable. In addition, using a faster hard disk can improve the performance of AOF.

Turn on AOF

By default, Redis does not enable AOF (append only file) persistence, which can be enabled by making the following configuration in the configuration file:

When turned on, every time Redis executes a write command, the command is written to the AOF file on the hard disk. The save path of AOF file is the same as that of RDB file, both are configured through the dir parameter. The default file name is: appendonly.aof, which can be modified by configuring the appendonlyfilename parameter, for example:

Implementation of AOF persistence

AOF records write commands executed by Redis in plain text, such as the following command with AOF persistence turned on:

Then look at the D:\ Redis_x64_321\ appendonly.aof file:

The content of the file is exactly the content of the command that Redis has just executed, and the format of the content will not be described.

AOF file rewriting

AOF files are recognizable plain text, and their contents are Redis standard commands.

AOF logs are not generated exactly as requested by the client. For example, the command INCRBYFLOAT is recorded as a SET record when recording AOF logs, because floating point operations may vary on different systems, so in order to prevent the same log from generating different data sets on different systems, only the results of the operation are recorded through SET.

Each write command generates a log, and the AOF file will be large.

AOF rewriting is a re-generation of an AOF file, and a record operation is performed only once in a new AOF file, unlike an old file, which may record multiple operations on the same value. Its generation process is similar to RDB, which is also a process of fork, which traverses the data directly and writes to a new temporary AOF file. During the process of writing to a new file, all write logs are still written to the old AOF file and recorded in the memory buffer. When the refinish operation is complete, the logs in all buffers are written to a temporary file at once. Then call the atomic rename command to replace the old AOF file with the new AOF file.

Command: BGREWRITEAOF, we should often call this command to rewrite.

=

Suppose Redis executes the following command:

It would be stupid if all these commands were written to the AOF file, because the first two commands would be overwritten by the third command, so the AOF file doesn't need to save the first two commands at all, which is exactly what Redis does. The process of deleting useless commands in the AOF file is called "AOF rewriting". AOF rewriting can be configured in the configuration file, and AOF rewriting is performed automatically when the configuration conditions are met. The configuration is as follows:

Auto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mb

The first line means that the current AOF file will be rewritten again when the size of the current AOF file exceeds that of the last AOF file, based on the size of the AOF file at startup if it has not been overridden before.

The second line means that the AOF file is rewritten only when the size of the AOF file is larger than 64MB, because if the AOF file is already small, it doesn't hurt to have several invalid commands.

These two configuration items are usually used together.

We can also manually execute the BGREWRITEAOF command and actively ask Redis to rewrite the AOF file:

View the current AOF file after executing the rewrite command:

As you can see, invalid commands such as set k v1 are no longer recorded in the file.

Synchronize hard disk data

Although AOF records the commands executed every time it changes the contents of the database, the contents of the AOF file are not really written to the hard disk because of the hard disk cache of the operating system itself. by default, the operating system synchronizes the data in the hard disk cache to the hard disk every 30 seconds, but in order to prevent data loss caused by abnormal exit of the system. We can also configure the frequency of this synchronization in the Redis configuration file:

# appendfsync alwaysappendfsync everysec# appendfsync no

The first line indicates that every time AOF writes a command, it performs a synchronous operation, which is the safest and slowest way.

The second line indicates that a synchronization operation occurs every second, which is generally sufficient

The third line indicates that synchronization does not take the initiative, which is the most insecure way.

Options:

1 、 appendfsync no

When appendfsync is set to no, Redis does not actively call fsync to synchronize AOF log contents to disk, so it all depends on the debugging of the operating system. For most Linux operating systems, fsync is performed every 30 seconds to write the data in the buffer to disk.

2 、 appendfsync everysec

When appendfsync is set to everysec, Redis makes a fsync call every other second by default to write the data in the buffer to disk. But when the fsync call takes more than 1 second this time. Redis will adopt the strategy of delaying fsync and wait one more second. That is, the fsync is performed two seconds later, and this time the fsync will take place no matter how long it takes to execute. At this point, because the file descriptor is blocked during fsync, the current write operation is blocked. So, the conclusion is: in the vast majority of cases, Redis will fsync every other second. In the worst-case scenario, a fsync operation occurs every two seconds. This operation is called group commit in most database systems, which combines the data of multiple write operations and writes the log to disk at one time.

3 、 appednfsync always

When appendfsync is set to always, fsync is called once for each write operation, so the data is the safest, and of course, because fsync is executed every time, its performance is also affected.

Appendfsync everysec is recommended (default)

Snapshot mode and AOF mode can be turned on at the same time to complement each other.

Third, the difference between the two

RDB persistence means that the snapshot of the dataset in memory is written to disk within a specified time interval. The actual operation is a child process of fork, which first writes the dataset to a temporary file, and then replaces the previous file and stores it with binary compression.

AOF persistence records every write and delete operation processed by the server in the form of log, query operation is not recorded, it is recorded in the form of text, and you can open the file to see the detailed operation record.

IV. Advantages and disadvantages of both

What are the advantages of RDB?

1)。 Once you use this approach, your entire Redis database will contain only one file, which is perfect for file backup. For example, you might plan to archive data for the last 24 hours every hour, as well as data for the last 30 days once a day. Through this backup strategy, in the event of a catastrophic failure of the system, we can recover very easily.

2)。 RDB is a very good choice for disaster recovery. Because we can easily compress a single file and then transfer it to other storage media.

3)。 Maximize performance. For the Redis service process, at the beginning of persistence, the only thing it needs to do is to fork the child process, and then the child process completes the persistence work, which can greatly prevent the service process from performing IO operations.

4)。 Compared with the AOF mechanism, if the dataset is large, the RDB startup efficiency will be higher.

What are the disadvantages of RDB?

1)。 If you want to ensure high availability of data, that is, to avoid data loss as much as possible, then RDB will not be a good choice. Because once the system goes down before timing persistence, data that has not been written to disk before will be lost.

2)。 Because RDB assists in data persistence through the help child process, if the dataset is large, it may cause the entire server to go out of service for hundreds of milliseconds, or even 1 second.

What are the advantages of AOF?

1)。 This mechanism can bring higher data security, that is, data persistence. Three synchronization strategies are provided in Redis, that is, synchronization per second, synchronization per modification, and async. In fact, synchronization per second is also done asynchronously, and its efficiency is also very high. The only difference is that once the system downtime occurs, then the modified data within this second will be lost. Every time we modify synchronization, we can think of it as synchronous persistence, that is, every data change that occurs is immediately recorded to disk. It can be predicted that this approach is the least efficient. As for non-synchronization, needless to say, I think everyone can understand it correctly.

2)。 Because this mechanism uses append mode to write log files, even if there is a downtime in the writing process, it will not destroy the content that already exists in the log file. However, if we only write half the data this time and the system crashes, don't worry, we can use the redis-check-aof tool to help us solve the data consistency problem before the next startup of Redis.

3)。 If the log is too large, Redis can automatically enable the rewrite mechanism. That is, Redis constantly writes the modified data to the old disk file in append mode, and Redis also creates a new file to record which modification commands were executed during this period. Therefore, data security can be better guaranteed when rewrite switching is carried out.

4)。 AOF contains a clearly formatted, easy-to-understand log file for recording all changes. In fact, we can also complete the reconstruction of the data through this file.

What are the disadvantages of AOF?

1)。 For the same number of datasets, AOF files are usually larger than RDB files. RDB recovers large datasets faster than AOF does.

2)。 Depending on the synchronization strategy, AOF tends to be slower than RDB in terms of efficiency. In short, the efficiency of synchronization per second policy is relatively high, and the efficiency of synchronization disable policy is as efficient as that of RDB.

The standard of choice between the two is to see whether the system is willing to sacrifice some performance for higher cache consistency (aof), or is willing to write frequently, do not enable backup in exchange for higher performance, and then do backup (rdb) when save is run manually. Rdb is even more eventually consistent.

5. Common configurations

RDB persistent configuration

Redis dump a snapshot of the dataset to the dump.rdb file. In addition, we can also modify the frequency of dump snapshots of the Redis server through the configuration file. After opening the 6379.conf file, we search save and see the following configuration information:

Save 9001 # after 900s (15 minutes), if at least one key changes, the dump memory snapshot.

Save 30010 # after 300 seconds (5 minutes), if at least 10 key changes, the dump memory snapshot.

Save 60 10000 # after 60 seconds (1 minute), if at least 10000 key changes, the dump memory snapshot.

AOF persistent configuration

There are three synchronization methods in the configuration file of Redis, which are:

Appendfsync always # is written to the AOF file every time a data modification occurs.

Appendfsync everysec # synchronizes once per second, which is the default policy for AOF.

Appendfsync no # is never synchronized. Efficient but the data is not persisted.

The above is the whole content of this article, I hope it will be helpful to your study, and I also hope that you will support it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.