Shulou(Shulou.com)06/02 Report--

What is the principle of Redis persistence? aiming at this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible way.

Redis provides the ability to automatically persist data to the hard disk on a regular basis, including RDB and AOF, which have their own advantages and disadvantages, respectively, and can work together to ensure the stability of the data.

RDB

Save a snapshot of the data to a RDB file for persistence. The RDB operation is similar to Mysql Dump.

Execution mode

Save . Synchronous operation, which blocks Redis.

Bgsave . Call fork () of linux, and then use the new thread to perform the copy. However, Redis is also blocked during fork, but the blocking time is usually very short.

Automatically save. The automatic saving trigger mechanism is set in the Redis configuration file, which can be customized and modified, and the operation principle is the same as bgsave.

Comparison between save and bgsave

Note:

If there is more than one Redis running on the machine, you need to configure the RDB file name, otherwise the RDB files of multiple Redis will overwrite each other.

In addition to the above three execution methods, RDB files are also generated in the following cases:

When the master and slave copy in full, the host generates a RDB file.

Debug reload in Redis provides a debug-level restart, a restart without emptying memory, which also triggers the generation of RDB files.

When shutdown is executed, the generation of the RDB file is triggered.

Shortcomings of RDB

Full data storage, time-consuming.

Although fork () uses the copy-on-write policy, it still consumes memory

Writing RDB files consumes a lot of IO performance.

AOF

With AOF persistence, Redis records every write request in a log file, and the AOF operation is similar to Mysql Binlog. Reduce the volume of AOF files through the AOF rewriting mechanism, thereby reducing the recovery time.

Execution mode

Always . Every write command from Redis is written to the system buffer, and then each write command uses fsync to "write" to the hard disk.

Everysec . The process is the same as always, except that the frequency of fsync is once per second. This is the default configuration of Redis. If the system goes down, about a second of data will be lost.

No . It is up to the operating system to decide when to flush from the system buffer to the hard disk.

AOF rewriting

In order to solve the problem of AOF file volume expansion, Redis provides AOF rewrite function: the Redis server can create a new AOF file to replace the existing AOF file, the new and old files save the database state is the same, but the new AOF file will not contain any space-wasting redundant commands, usually the volume will be much smaller than the old AOF file.

AOF rewriting mode

Bgrewriteaof (process similar to bgsave)

AOF rewrite configuration (similar to RDB auto-save)

> AOF rewriting does not require any read, write, analysis and other operations to the original AOF file. This function is achieved by reading the current database status of the server.

RDB vs AOF

Data loading at startup of Redis

Redis starts the data loading process:

When AOF persistence is enabled and an AOF file exists, the AOF file is loaded first.

When the AOF is closed or the AOF file does not exist, load the RDB file.

After loading the AOF/RDB file successfully, Redis starts successfully.

When there is an error in the AOF/RDB file, Redis fails to start and prints an error message.

Development of common problems in operation and maintenance fork operation

The actual cost of fork () is to copy the page table of the parent process and create a process descriptor for the child process, so it is generally faster.

> the larger the amount of memory, the longer the time; the physical machine is relatively fast, and the virtual machine is relatively slow.

Optimization method

Give priority to physical machines or virtualization technologies that efficiently support fork operations

Control the maximum available memory maxmemory of a Redis instance

Reasonable configuration of Linux memory allocation policy: vm.overcommit_memory=1. The default value is 0, which will cause Linux not to allocate enough memory when it finds that there is insufficient memory, thus causing fork blocking.

Reduce the frequency of fork. For example, relax the automatic trigger time of AOF rewriting or reduce unnecessary master-slave full replication.

Out of process overhead

CPU . RDB and AOF file generation is CPU-intensive. Do not bind Redis processes to a CPU to prevent single core overload; and Redis is not deployed with CPU-intensive applications.

Memory. Fork memory overhead, copy-on-write.

Hard drive. Writing of AOF and RDB files. Can be combined with iostat and iotop for analysis.

Optimization method

Do not deploy with high hard disk load services: storage services, message queues, etc.

Configure no-appendfsync-on-rewrite=yes. This reduces memory overhead by avoiding AOF append operations during AOF rewriting (the main thread only writes data to the buffer).

> however, if Redis goes down during AOF rewriting, a maximum of 30 seconds of data will be lost under the default configuration of Linux. If data loss cannot be tolerated, no-appendfsync-on-rewrite configures no; to set to yes if the application system cannot tolerate delay and can tolerate a small amount of data loss.

Determine the disk type based on the number of writes: for example, ssd

Multi-instance persistent file directories on a single machine can be considered as separate disks, or a mechanism similar to cgroups can be used for rational allocation of hard disk resources.

AOF additional blocking

For example, in AOF's everysec policy, the main thread compares the time of the last fsync, and if it is more than two seconds since the last fsync, it will cause the main thread to block (waiting for the synchronization thread to complete synchronization).

Daily developers can use the info persistence command to check the number of times AOF blocking occurred in history; however, if you need to know when AOF additional blocking occurs, you need to check the Redis log.

When sending AOF append blocking, the log is as follows: > Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.

The answer to the question about the principle of Redis persistence is shared here. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.