Shulou(Shulou.com)05/31 Report--

This article focuses on "how to configure high availability and persistence in redis". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let the editor take you to learn how to configure high availability and persistence in redis.

1. Redis High availability 1. Overview of Redis High availability

   in web servers, high availability refers to the time that the server can be accessed normally, which is measured by how long it takes to provide normal services (99.9%, 99.99%, 99.999%, etc.). [related recommendation: Redis video tutorial]

However, in the context of Redis, the meaning of high availability seems to be broader. In addition to ensuring the provision of normal services (such as master-slave separation, fast disaster recovery technology), we also need to consider the expansion of data capacity, data security will not be lost, and so on.

2. Redis high availability strategy

In Redis, the technologies to achieve high availability include persistence, master-slave separation, sentinels, and clustering.

The high availability policy states that persistence persistence is the simplest method of high availability (sometimes not even classified as a highly available means). Its main function is to back up the data, that is, to store the data on the hard disk to ensure that the data will not be lost due to the exit of the process. Master-slave replication is the basis of high-availability Redis. Sentinel and cluster achieve high availability on the basis of master-slave replication. Master-slave replication mainly realizes multi-machine backup of data, as well as load balancing and simple fault recovery for read operations. Defects: failure recovery can not be automated, write operations can not be load balanced, and storage capacity is limited by a single machine. On the basis of master-slave replication, Sentinel realizes automatic fault recovery. Defect: the write operation cannot be load balanced, and the storage capacity is limited by a single machine. Cluster through the cluster, Redis solves the problem that the write operation can not be load balanced and the storage capacity is limited by a single machine, and achieves a relatively perfect high availability solution. 2. Redis persistence 1. Redis persistence function

   Redis is an in-memory database, and the data is stored in memory. In order to avoid the permanent loss of data after the abnormal exit of the Redis process caused by the power outage of the server, it is necessary to save the data in Redis in some form (data or commands) from memory to the hard disk on a regular basis; when Redis restarts next time, use persistent files to achieve data recovery. In addition, for disaster backup, persistent files can be copied to a remote location.

2. Two ways of Redis persistence

RDB persistence

The principle is that the database records of Redis in memory are saved to disk regularly.

AOF persistence (append only file)

The principle is to write the operation log of Redis to a file in an appended way, similar to MySQL's binlog.

Because AOF persistence is more real-time, that is, less data is lost when the process exits unexpectedly, AOF is the mainstream persistence method at present, but RDB persistence still has the opportunity to show its ability.

3. RDB persistence

   RDB persistence means to generate a snapshot of the data in the current process in memory and save it to the hard disk within a specified time interval (so it is also known as snapshot persistence), using binary compression storage, and the saved file suffix is rdb;. When Redis restarts, the snapshot file recovery data can be read.

3.1 trigger condition

There are two kinds of triggers for RDB persistence: manual trigger and automatic trigger.

3.1.1 Manual trigger

Both the save command and the bgsave command can generate RDB files.

The save command blocks the Redis server process until the RDB file is created, and the server cannot process any command requests during the Redis server blocking.

The bgsave command fork () a child process, which is responsible for creating the RDB file, and the parent process (that is, the Redi main process) continues to process the request.

During the execution of the bgsave command, only the save child process will block the server, while for the save command, the whole process will block the server, so save has been basically abandoned, and the online environment should eliminate the use of save.

3.1.2 automatic trigger

When RDB persistence is automatically triggered, Redis also chooses bgsave over save for persistence.

3.2 configuration mode

Set by modifying the configuration file: save m n

The most common case of automatic trigger is through save m n in the configuration file, specifying that bgsave will be triggered when n changes occur in m seconds.

[root@localhost ~] # vim / etc/redis/6379.conf # 219 lines, when any of the following three save conditions are satisfied, it will cause bgsave to call save 9001 # # when the time is 900s, if the redis data has changed at least once, then execute bgsavesave 30010 # when the time is 300s, if the redis data has changed at least 10 times, then execute bgsavesave 60 10000 # # when the time has reached 60 seconds If the redis data has changed at least 10000 times, execute the bgsave##254 line, specify the RDB file name dbfilename dump.rdb##264 line, specify the dir / var/lib/redis/6379##242 line where the RDB file and AOF file are located, and whether to turn on other automatic trigger mechanisms of RDB file compression rdbcompression yes3.3

In addition to save m n, there are other situations that trigger bgsave:

In the master-slave replication scenario, if the slave node performs a full copy operation, the master node executes the bgsave command and sends the rdb file to the slave node.

Rdb persistence is automatically performed when the shutdown command is executed.

3.4 execute the process

The Redis parent process first determines whether save, or a child of bgsave/bgrewriteaof, is currently executing, and if so, the bgsave command returns directly. Bgsave/bgrewriteaof 's child processes cannot be executed at the same time, mainly based on performance considerations; two concurrent child processes perform a large number of disk writes at the same time, which may cause serious performance problems.

The parent process executes the fork operation to create the child process, in which the parent process is blocked and Redis cannot execute any commands from the client.

After the parent process fork, the bgsave command returns "Background saving started" information which no longer blocks the parent process and can respond to other commands

The child process creates the RDB file, generates a temporary snapshot file according to the memory snapshot of the parent process, and then atomic replaces the original file.

The child process sends a signal to the parent process that it is complete, and the parent process updates the statistics

3.5 load at startup

The loading of    RDB files is performed automatically when the server starts, and there are no special commands. However, because AOF has a higher priority, when AOF is turned on, Redis will load the AOF file first to recover the data; only when AOF is turned off will the RDB file be detected and loaded automatically when the Redis server starts. The server is blocked during the loading of the RDB file until the loading is complete.

When    Redis loads the RDB file, the RDB file is checked. If the file is corrupted, an error is printed in the log and Redis startup fails.

4. AOF persistence

   RDB persistence writes process data to a file, while AOF persistence records every write and delete command executed by Redis in a separate log file, and query operations are not recorded. When Redis restarts, execute the commands in the AOF file again to recover the data.

Compared with RDB,    has better real-time performance, so AOF has become the mainstream persistence scheme.

4.1 enable AOF

By default, Redis server enables RDB and disables AOF;. To enable AOF, you need to configure it in the configuration file.

[root@localhost ~] # vim / etc/redis/6379.conf # 700line, modify, open AOFappendonly yes##704 line, specify AOF file name appendfilename "appendonly.aof" # # 796line, whether to ignore the last instruction aof-load-truncated yes [root@localhost ~] # / etc/init.d/redis_6379 restartStopping... Redis stoppedStarting Redis server...4.2 execution flow

Since each write command of Redis needs to be recorded, AOF does not need to be triggered, so the execution flow of AOF is described below.

The execution process of AOF includes:

Command append (append): appends the write command of Redis to the buffer aof_buf

File write (write) and file synchronization (sync): synchronize the contents of aof_buf to the hard disk according to different synchronization strategies

File rewriting (rewrite): rewrite AOF files regularly to achieve the purpose of compression.

4.2.1 Command append (append)

Redis appends the command to the buffer first instead of writing directly to the file, mainly to avoid writing to the hard disk directly every time there is a write command, resulting in the hard disk IO called the bottleneck of the Redis load.

The appended format of the command is the protocol format requested by the Redis command, which is a plain text format, which has the advantages of good compatibility, strong readability, easy to deal with, simple operation and avoiding secondary overhead. In the AOF file, except for the select command used to specify the database (for example, select 0 is the selected database 0) is added by Redis, the other write commands are sent from the client.

4.2.2 File write (write) and file synchronization (sync)

Redis provides a variety of file synchronization policies for AOF cache, which involves the write and fsync functions of the operating system, as described below:

In order to improve the efficiency of file writing, in modern operating systems, when users call the write function to write data to a file, the operating system usually temporarily stores the data in a memory buffer. When the buffer is filled or exceeds the specified time limit, the buffer data is actually written to the hard disk. Although this operation improves efficiency, it also brings security problems: if the computer stops, the data in the memory buffer will be lost; therefore, the system also provides synchronization functions such as fsync and fdatasync, which can force the operating system to immediately write the data in the buffer to the hard disk, thus ensuring the security of the data.

4.2.3 three synchronization methods

There are three ways to synchronize files in the AOF cache, which are configured by modifying line 729 of / etc/redis/6379.conf.

4.2.3.1 appendfsync always

Immediately after the command is written into aof_buf, the system fsync operation is called to synchronize to the AOF file, and the thread returns after the fsync is completed. In this case, every time there are write commands have to be synchronized to the AOF file, the hard disk IO has become a performance bottleneck, Redis can only support about a few hundred TPS writes, seriously reducing the performance of Redis; even if you use solid state disk (SSD), you can only handle tens of thousands of commands per second, and will greatly reduce the life of SSD.

4.2.3.2 appendfsync no

After the command is written into aof_buf, the system write operation is called without fsync synchronization of AOF files; synchronization is loaded by the operating system, and the synchronization period is usually 30 seconds. In this case, the time of file synchronization is uncontrollable, and there will be a lot of data accumulated in the buffer, so the data security can not be guaranteed.

4.2.3.3 appendfsync everysec (recommended)

The write operation of the system is called after the command is written into aof_buf, and the thread returns after the write is completed: the fsync synchronization file operation is called once per second by a special thread. Everysec is a compromise between the above two strategies, a balance between performance and data security, and once the default configuration of Redis, which is also our recommended configuration.

4.2.4 File rewriting (rewrite)

With the passage of time, the Redis server executes more and more write commands, and the AOF file will become larger and larger; too large AOF files will not only affect the normal operation of the server, but also cause the data recovery time to be too long.

File rewriting refers to regularly rewriting AOF files to reduce the size of AOF files. It should be noted that AOF rewriting converts the data in the Redis process into write commands and synchronizes it to the new AOF file; no read or write operations are performed on the old AOF file.

Another thing to note about file rewriting is that for AOF persistence, file rewriting is highly recommended, but not necessary; even if there is no file rewriting, data can be persisted and imported when Redis starts; so in some reality, automatic file rewriting is turned off and scheduled tasks are executed regularly at a certain time of day.

4.2.4.1 reasons for compression

The reason why file rewriting can compress AOF files is that:

Expired data is no longer written to the file.

Invalid commands are no longer written to the file: for example, some data is set repeatedly (set mykey v1jime set mykey v2), some data is deleted (set myset v1jindel myset), and so on.

Multiple commands can be merged into one: for example, sadd myset v1 myset v2 add myset v2 myset v3 can be merged into sadd myset v1 v2 v3.

As can be seen from the above reasons, because AOF executes fewer commands after rewriting, file rewriting can not only reduce the space occupied by files, but also speed up recovery.

4.2.4.2 trigger for file rewriting

File rewriting is divided into manual trigger and automatic trigger:

Manual trigger: invoke the bfrewriteaof command directly, which is somewhat similar to bgsave in that the fork process does specific work and is blocked only during fork.

Automatic trigger: automatically execute bgrewriteaof by setting the auto-aof-rewrite-min-size option and the auto-aof-rewrite-percentage option. AOF rewriting, that is, the bgrewriteaof operation, is triggered automatically only if both the auto-aof-rewrite-min-size and auto-aof-rewrite-percentage options are met.

The automatically triggered configuration is located on lines 771 and 772 of / etc/redis/6379.conf

Auto-aof-rewrite-percentage 100

A bgrewriteaof operation occurs when the current AOF file size (that is, aof_current_size) is twice the AOF file size (aof_base_size) at the time of the last log rewrite

Auto-aof-rewrite-min-size 64mb

The minimum value for the current AOF file to execute the bgrewriteaof command to avoid frequent bgrewriteaof due to the small file size when starting Redis

4.2.4.3 process for file rewriting

The process of file rewriting is as follows:

The Redis parent process first determines whether there is a child process executing bgsave/bgrewriteaof; if so, the bgrewriteaof command returns directly, and if there is a bgsave command, it waits for the bgsave execution to be completed.

The parent process performs the fork operation to create the child process, in which the parent process is blocked.

After the parent process fork, the bgrewriteaof command returns "Background append only file rewrite started" information which no longer blocks the parent process and can respond to other commands. All Redis write commands are still written to the AOF buffer and synchronized to the hard disk according to the appendfsync policy to ensure the correctness of the original AOF mechanism.

Because fork operations use write-time replication technology, child processes can only share in-memory data during fork operations. Because the parent process is still responding to commands, Redis uses the AOF rewrite buffer (aof_rewrite_buf) to save this data to prevent it from being lost during the generation of the new AOF file. That is, during the execution of bgrewriteaof, the write command of Redis is appended to both aof_buf and aof_rewrite_buf buffers.

According to the memory snapshot, the child process writes to the new AOF file according to the command merge rules.

After the child process has finished writing the new AOF file, it signals to the parent process that the parent process updates the statistics, which can be viewed through info persistence.

The parent process writes the data from the AOF rewrite buffer to the new AOF file, which ensures that the database state saved by the new AOF file is consistent with the current state of the server.

Replace the old file with the new AOF file and rewrite the text into AOF.

With regard to the process of file rewriting, there are two points to pay special attention to:

The rewrite is done by the parent process, the child process of fork

Write commands executed by Redis during rewriting need to be appended to a new AOF file, for which Redis introduces a cache such as aof_rewrite_buf

4.3 load at startup

When AOF is enabled, Redis will first load the AOF file to recover the data when it starts; only when AOF is turned off will the RDB file be loaded to recover the data.

When AOF is turned on, but the AOF file does not exist, it will not be loaded even if the RDB file exists.

When Redis loads the AOF file, it verifies the AOF file. If the file is corrupted, an error will be printed in the log and Redis startup fails. However, if the end of the AOF file is incomplete (the tail of the file is easily incomplete due to sudden machine downtime, etc.), and the aof_load_truncated parameter is enabled, a warning will be output in the log. Redis ignores the tail of the AOF file and starts successfully. The aof_load_truncated parameter is enabled by default.

5. Advantages and disadvantages of RDB and AOF

RDB persistence

Advantages: RDB files are compact, small, fast network transmission, suitable for full replication; recovery speed is much faster than AOF. Of course, one of the most important advantages of RDB over AOF is that it has relatively little impact on performance.

Disadvantages: the well-known disadvantage of RDB files is that the persistence of its data snapshots determines that real-time persistence cannot be done. Today, when data is becoming more and more important, the loss of a large amount of data is often unacceptable, so AOF persistence has become the mainstream. In addition, RDB files need to meet specific formats and have poor compatibility (for example, the old version of Redis is not compatible with the new version of RDB files).

For RDB persistence, on the one hand, the Redis main process will block when bgsave performs fork operations, on the other hand, the child process will also bring IO pressure to write data to the hard disk.

AOF persistence

Corresponding to RDB persistence, the priority of AOF is to support second persistence and good compatibility, while the disadvantages are large files, slow recovery, and great impact on performance.

For AOF persistence, the frequency of writing data to the hard disk is greatly increased (seconds under everysec policy), IO pressure is greater, and may even add blocking problems to AOF.

The rewriting of AOF files is similar to RDB's bgsave, which can cause blocking in fork and IO pressure problems in child processes. Relatively speaking, because AOF writes data to the hard disk more frequently, it has a greater impact on the performance of the Redis main process.

Generally speaking, it is recommended to disable the automatic rewriting feature of AOF and set the scheduled task in the rewrite operation to be carried out in the early hours of the morning when the traffic is low, so as to reduce the impact of AOF on the performance of the main process and the reading and writing pressure on IO.

At this point, I believe you have a deeper understanding of "how to configure high availability and persistence in redis". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.