Shulou(Shulou.com)05/31 Report--

This article will explain in detail the example analysis of the disk persistence mechanism in Redis. Xiaobian thinks it is quite practical, so share it with you for reference. I hope you can gain something after reading this article.

text

Redis is a database server based on K-V storage. The following describes the internal structure of Redis database and the storage form of K-V, which helps us to understand the persistence mechanism of Redis more easily.

1. Redis database structure

A stand-alone Redis server has 16 databases (0-15) by default, and the default database is database 0. You can switch databases by using SELECT command.

Each database in Redis is represented by a redis.h/redisDb structure that records the key space of a single Redis database, the expiration time of all keys, the keys in blocked and ready states, the database number, and so on.

typedef struct redisDb { //Database key space, storing all key pairs dict *dict in the database; //Expiration time of keys, dictionary keys, dictionary values for expiration events UNIX timestamp dict *expires; //Keys blocked dict *blocking_keys; //Keys that can be unblocked dict *ready_keys; //Keys monitored by WATCH command dict *watched_keys; struct evictionPoolEntry *eviction_pool; //database number int id; //average TTL of database keys, statistics long long avg_ttl;} redisDb;

Since Redis is a database of key-value pairs, its database itself is also a dictionary, and the corresponding structure is redisDb. dict refers to a dictionary of key-value pairs. Its key is a string object, and its value can be any Redis type object, including string, list, hash table, set and ordered set. expires points to a dictionary that records the expiration time of a key. Its key is the database key in dict. Its value is the expiration timestamp of this database key. This value is represented by long long type.

2. RDB Persistence

RDB persistence (also known as snapshot persistence) refers to saving a snapshot of data in memory to disk with the file suffix.rdb. The rdb file is a compressed binary file that can be read from the rdb snapshot file to recover data when Redis restarts. The core of RDB function is rdbSave and rdbLoad. The former is used to generate RDB file and save it to disk, while the latter is used to reload the data in RDB file into memory:

RDB file is a single file of full data, very suitable for data disaster recovery backup and recovery, through the RDB file recovery database time is relatively short, usually 1G snapshot file loaded into memory only about 20s. Redis provides manual trigger save, automatic save interval two RDB file generation methods, the following first describes the RDB creation and loading process.

2.1. RDB creation and loading

2.1.1. Manual trigger save

Redis provides two commands for generating RDB files, SAVE and BGSAVE. There are two ways to trigger Redis to perform RDB backup. One is to trigger snapshot generation manually through SAVE command and BGSAVE command. The other is to configure save time and write times, and Redis automatically triggers save operation according to conditions.

1. SAVE command

SAVE is a synchronous command that blocks the Redis server process until the RDB file is created. The server cannot process any other command requests while the server process is blocked.

client command

127.0.0.1:6379> SAVEOK

server-side log

6266:M 15 Sep 2019 08:31:01.258 * DB saved on disk

After executing the SAVE command, Redis performs the SAVE operation on the server process (PID 6266), blocking Redis client request processing for the duration of this operation.

2. BGSAVE command

BGSAVE is an asynchronous command. Unlike SAVE, which blocks the server process directly, BGSAVE spawns a child process that creates the RDB file. The server process (parent process) continues to process the client's commands.

client command

127.0.0.1:6379> BGSAVEBackground saving started

server-side log

6266:M 15 Sep 2019 08:31:22.914 * Background saving started by pid 62836283:C 15 Sep 2019 08:31:22.915 * DB saved on disk6266:M 15 Sep 2019 08:31:22.934 * Background saving terminated with success

Through the log output by the server, it can be found that Redis creates (forks) a child process (PID 6283) for the BGSAVE command separately in the server process (PID 6266), and the child process completes the RDB saving process in the background, notifies the parent process after the operation is completed, and then exits. During the entire process, the server process spends only a small amount of time creating child processes and processing child process semaphores, and the rest of the time on standby.

BGSAVE is the mainstream way to trigger RDB persistence. The following is the flow of BGSAVE command to generate snapshot:

The client initiates the BGSAVE command, and the Redis main process judges whether there is a child process currently executing backup, and if there is, it returns directly.

The parent process forks a child process (the fork process causes blocking). This process can be checked by using the info stats command to check the latest_fork_usec option to see the time spent by the last fork operation in microseconds.

After the parent process fork is completed, it will return a message prompt of Background saving started. At this time, the fork blocking is lifted.

The child process created by fork starts to generate temporary snapshot files based on the memory data of the parent process, and then replaces the original file

After the child process is backed up, it sends completion information to the parent process, which updates statistics.

3. Comparison of SAVE and BGSAVE

Command SAVEBGSAVEIO Type Synchronous Asynchronous Whether blocking occurs during full fork Complexity O(n)O(n) Advantages No extra memory consumption No blocking of client disadvantages Blocking client fork Child processes consumes memory

2.1.2. Automatic trigger save

Because the BGSAVE command can be executed without blocking the server process, the Redis configuration file redis.conf provides a save option that causes the server to automatically execute the BGSAVE command at intervals. The save option allows the user to set multiple save conditions, and if any of these conditions is met, the server executes the BGSAVE command. The Redis configuration file redis.conf is configured with the following three storage conditions by default:

save 900 1save 300 10 save 60 10000

The BGSAVE command is automatically executed if any of the following three conditions are met:

The server has modified the database at least once in 900 seconds.

The server made at least 10 changes to the database within 300 seconds.

The server made at least 10000 changes to the database in 60 seconds.

Redis server periodically operates the serverCron function, which executes every 100 milliseconds. One of its tasks is to check whether the save condition set by the save option is met, and if so, automatically execute the BGSAVE command.

2.1.3. Start automatic loading

Unlike using SAVE and BGSAVE commands to create RDB files, Redis does not specifically provide commands for loading RDB files, and the loading process for RDB files is done automatically at Redis server startup. Redis automatically loads RDB files via the rdbLoad function whenever it detects the existence of RDB files in the specified directory at startup.

Below are the logs printed when the Redis server starts up, and the second log from the bottom is printed after successfully loading the RDB file.

$ redis-server /usr/local/etc/redis.conf6266:C 15 Sep 2019 08:30:41.830 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=6266, just started6266:C 15 Sep 2019 08:30:41.830 # Configuration loaded6266:M 15 Sep 2019 08:30:41.831 * Increased maximum number of open files to 10032 (it was originally set to 256).6266:M 15 Sep 2019 08:30:41.832 # Server initialized6266:M 15 Sep 2019 08:30:41.833 * DB loaded from disk: 0.001 seconds6266:M 15 Sep 2019 08:30:41.833 * Ready to accept connections

Because AOF files belong to incremental write command backups and RDB files belong to full data backups, the update frequency is higher than that of RDB files. So if the Redis server has AOF persistence enabled, the server will use AOF files to restore the database state first; only when AOF persistence is turned off, the server will use RDB files to restore the database state first.

2.2. File structure of RDB

RDB files are compressed binary files. Here are some details about RDB files.

2.2.1. storage path

Both the SAVE command and the BGSAVE command back up only the current database. The backup file name defaults to dump.rdb. The backup file name dbfilename xxx.rdb can be modified through the configuration file. Backup file directories and RDB file names can be viewed by:

$ redis-cli -h 127.0.0.1 -p 6379127.0.0.1:6379> CONFIG GET dir1) "dir"2) "/usr/local/var/db/redis"127.0.0.1:6379> CONFIG GET dbfilename1) "dbfilename"2) "dump.rdb"

The storage path of RDB files can be configured before startup or dynamically by command.

Configuration item: Specify directory via dir configuration, dbfilename specifies file name

Dynamic designation: Redis can also dynamically modify the RDB storage path after startup, which is very useful when the disk is damaged or insufficient space. The execution command is:

config set dir {newdir}config set dbfilename {newFileName} About "Example analysis of disk persistence mechanism in Redis" This article is shared here, I hope the above content can be of some help to everyone, so that you can learn more knowledge, if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.