Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "what is the underlying principle of RDB technology in Redis". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Summary of premises

Redis is an in-memory database service for a key-value pair, which usually contains any non-empty database. Any Kmurv can be stored in each non-empty key-value database. The basic structure is shown in the following figure:

The strong performance of Redis is largely due to the fact that it stores all the data in memory. In order to ensure that the data is not lost after Redis is rebooted, it is necessary to synchronize the data from memory to the hard disk in some form. This process is persistence.

We know that the cached data in redis is stored in memory, and once the service fails, it will lead to the loss of data in memory, so we need a data persistence scheme to write the data in redis memory to disk, and when redis restarts, the data can be recovered from disk.

The structure of Redis server

There is a problem here, because Redis is an in-memory database, if it stores the data directly in memory, but if you do not consider persisting the data stored in memory to the hard disk, once the server process exits, the data in the database will also disappear.

There are mainly two kinds of database persistence mechanisms, one is RDB mechanism, the other is AOF mechanism, which has been introduced in the previous article.

If you are interested, you can take a look, and this article focuses on the RDB mechanism.

RDB persistence mode

RDB persistence means that the snapshot of the dataset in redis memory is written to disk within a specified time interval. The implementation principle is that the redis service first fork a child process within the specified time interval, and the child process writes the dataset into a temporary file. After writing successfully, it replaces the previous file, uses binary compression storage, and generates dump.rdb files to store on disk.

RDB mechanism

Redis provides RDB persistence, which keeps Redis's in-memory database state on disk to avoid accidental data loss.

The RDB persistence mechanism can be performed manually or periodically according to the server configuration, which can save a snapshot of the data at a certain point in time to a RDB file.

RDB advantage

Once you use this approach, your entire Redis database will contain only one file, which is perfect for file backup. For example, you might plan to archive data for the last 24 hours every hour, as well as data for the last 30 days once a day. Through this backup strategy, in the event of a catastrophic failure of the system, we can recover very easily.

RDB is a very good choice for disaster recovery. Because we can easily compress a single file and then transfer it to other storage media.

Maximize performance. For the Redis service process, at the beginning of persistence, the only thing it needs to do is to fork the child process, and then the child process completes the persistence work, which can greatly prevent the service process from performing IO operations.

Compared with the AOF mechanism, if the dataset is large, the RDB startup efficiency will be higher.

RDB disadvantage

If you want to ensure high availability of data, that is, to avoid data loss as much as possible, then RDB will not be a good choice. Because once the system goes down before timing persistence, data that has not been written to disk before will be lost.

Because RDB assists in data persistence through the help child process, if the dataset is large, it may cause the entire server to go out of service for hundreds of milliseconds, or even 1 second.

RDB configuration rules

In the 6379.conf configuration file of redis:

Backup configuration parameters

Save

Save, synchronize the data in memory to the hard disk when the condition is met. The official factory configuration defaults to 1 change in 900 seconds, 10 changes in 300 seconds, and 10000 changes in 60 seconds, then write a snapshot of the data in memory to disk.

Save 9001 # after 900s (15 minutes), if at least one key changes, dump memory snapshot

Save 30010 # after 300 seconds (15 minutes), if at least 10 key changes, dump memory snapshot

Save 60 10000 # after 60 seconds (1 minute), if at least 10000 key changes, the dump memory snapshot

File configuration parameters

The default rdb file path is the current directory, and the file name is dump.rdb. You can modify the path and file name in the configuration file, dir and dbfilename.

# directory where snapshots are stored dir. / # rdb file storage path dbfilename dump.rdb # rdb file name compression configuration parameters

Whether to compress when making a mirror backup.

Rdbcompression yes # Redis enables compression by default. # yes: compression, but requires some cpu consumption. # no: no compression, more disk space is required.

"if automatic snapshots are not triggered, you need to perform manual snapshot operations on Redis, save and bgsave commands to manually snapshot, the two commands are:"

SAVE: snapshots taken by the main process block other requests.

BGSAVE: taking a snapshot through the help child process does not block other requests.

Note: because Redis uses fork to copy a copy of the current process, then the child process will occupy the same memory resources as the main process, such as 8 GB of memory for the main process, so when backing up, you must ensure that you have 16 GB of memory, otherwise virtual memory will be enabled, and the performance is very poor.

The procedure for snapshots is as follows:

Redis uses the fork function to make a copy of the current process (parent process) (child process)

The parent process continues to receive and process commands from the client, while the child process begins to write data in memory to temporary files on the hard disk

When the child process has written all the data, it replaces the old RDB file with the temporary file, and the snapshot operation is complete. (note: there will be a write command compression cache to record the operation when writing to the rdb file)

The operating system will use the copy-on-write policy when executing fork, that is, the parent and child processes share the same memory data at the moment when the fork function occurs. When the parent process wants to change a piece of data (such as executing a write command), the operating system will copy a copy of the data to ensure that the data of the child process will not be affected, so the new RDB file stores the memory snapshot data of the moment when fork is executed.

Through the above process, we can find that Redis will not modify the RDB file during the snapshot, and will replace the old file with the new one only after the snapshot ends, that is to say, the RDB file is complete at any time. This makes it possible to back up Redis databases by backing up RDB files on a regular basis.

Process compression analysis of snapshots:

RDB files are compressed (described above: you can configure the rdbcompression parameter to disable compression to save CPU footprint), so the space consumed will be less than the size of the data in memory, which is more convenient for transmission.

The read load process of the snapshot:

After Redis starts, it reads the RDB snapshot file and loads the data from the hard drive into memory. This time varies depending on the amount of data and structure and server performance. It usually takes 20 to 30 seconds to load a snapshot file with 10 million string type keys and the size of 1GB into memory.

Persistence is achieved through RDB. Once the Redis exits abnormally, all data changed since the last snapshot will be lost. This requires developers to control the possible data loss in an acceptable range by combining and setting automatic snapshot conditions according to the specific application situation. If the data is too important to bear any loss, consider using AOF for persistence.

Advantages and disadvantages of RDB

Advantages:

Suitable for large-scale data recovery.

If the business does not require high data integrity and consistency, RDB is a good choice.

Disadvantages:

The integrity and consistency of the data is not high because RDB may be down during the last backup.

The backup takes up memory because Redis creates a child process independently during the backup, writes data to a temporary file (where there is twice as much data in memory), and finally replaces the temporary file with the previous backup file.

Because RDB assists in data persistence through the help child process, if the dataset is large, it may cause the entire server to go out of service for hundreds of milliseconds, or even 1 second. (the main process is used for writing back and overwriting).

The standard chosen by both RDB and AOF (although there is no mention of AOF, it will be popularized in advance)

If the system is willing to sacrifice some performance for higher cache consistency (aof)

Or when you are willing to write frequently, do not enable backup in exchange for higher performance, wait until you run save manually, then do a backup (rdb).

Redis allows both AOF and RDB to be turned on at the same time, which not only ensures data security, but also makes backup and other operations very easy. After restarting Redis at this point, Redis uses the AOF file to recover the data, because AOF persistence may lose less data.

Summary

Redis enables RDB persistence by default. If you perform a specified number of write operations within a specified time interval, the data in memory will be written to disk.

RDB persistence is suitable for large-scale data recovery, but its data consistency and integrity are poor.

Redis needs to enable AOF persistence manually. By default, write logs are appended to the AOF file every second.

Therefore, it is more reasonable for Redis persistence and data recovery to be performed in the dead of night.

This is the end of the content of "what is the underlying principle of RDB technology in Redis". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.