Shulou(Shulou.com)05/31 Report--

Xiaobian to share with you the example analysis of the master-slave synchronization mechanism in Redis, I believe most people do not know much, so share this article for your reference, I hope you have a lot of gains after reading this article, let's go to understand it together!

Previous articles have analyzed the characteristics and core principles of redis in detail. Starting from this article, we will analyze and interpret the deployment structure and operation mode of redis. In a real production environment, we basically do not use single-node redis to provide services, at least in the master-slave structure of sentinel or cluster mode, to ensure the reliability of redis services. This article explains the master-slave synchronization mechanism of redis in detail.

Redis has two structural models:

1.1 master-slave replication

This replication structure of one master and N slaves has only one level of replication relationship, which is also the most commonly used form. Usually, redis that build sentries or cluster structures adopt this replication structure, which can ensure the availability of services through the replication relationship of one level of slave nodes and achieve master-slave switching in abnormal situations.

1.2 cascaded replication

The replication relationship of a cascade replication structure can have multiple levels, and the slave nodes of a master node can be the master nodes of subordinate slave nodes. Cascaded replication structure is relatively rare, and this structure can relieve the replication pressure of master node to some extent under the structure of multiple slave nodes.

Second, the establishment of Redis master-slave relationship

Master-slave synchronization of Redis begins with the command SLAVEOF host port, which establishes master-slave relationships, and SLAVEOF is used to dynamically modify the behavior of the replication function while Redis is running. By executing the SLAVEOF host port command, you can turn the current server into a slave server for the specified server. If the current server is already a slave to a master server, executing SLAVEOF host port causes the current server to stop synchronizing to the old master, discard the old dataset, and start synchronizing to the new master instead. In addition, executing the command SLAVEOF NO ONE on a slave server causes the slave server to turn off replication and transition from the slave server back to the master server without discarding the original synchronized data set. SLAVEOF NO ONE does not discard synchronized data sets. Without sentinels and clusters, slave servers can be used as new master servers when master servers fail, thus achieving uninterrupted operation.

The following figure shows the process of establishing the master-slave relationship:

Note:

According to the above execution process, there is a point to note here. When we execute the slaveof command on a node with an existing master-slave relationship, it will end the existing master-slave relationship and clear all the data under the node. This is a relatively threatening operation in the generation environment. Is there a safer way? When introducing the slavelof command above, it is mentioned that you can pass the NO ONE parameter, that is, execute the SLAVEOF NO ONE command. This command will only end the master-slave replication relationship and will not empty the data. It is relatively safe.

III. Data synchronization

After establishing the master-slave relationship, it is necessary to enter the process of master-slave data synchronization, which is mainly divided into three situations: full synchronization of data after establishing the master-slave relationship; command propagation stage after initialization synchronization; synchronization mode selection after abnormal interruption of master-slave relationship. There will be two scenarios of full synchronization and incremental synchronization.

3.1 total synchronization

When the slave node is started or disconnected and reconnects (the reconnect does not satisfy the incremental synchronization condition), SYNC command is sent to master database.

After receiving SYNC command, master node will start to save snapshot in background (i.e. RDB persistence, RDB will be triggered unconditionally in master-slave replication), and cache the command received during snapshot saving.

After the master node performs RDB persistence, it sends a snapshot RDB file to all slave nodes and continues to record the write commands executed during the snapshot sending.

The slave node receives the snapshot file, discards all old data (it empties all data), and loads the received snapshot.

After the master node sends the snapshot and the slave node loads the snapshot, the master node starts sending write commands in the buffer to the slave node.

The slave node completes loading the snapshot, starts receiving command requests, and executes write commands from the primary database buffer. (Completed from database initialization)

Each write command executed by the master node sends the same write command to the slave node, which receives and executes the received write command. (Command propagation operation, operation after slave node initialization is completed)

The full synchronization process is as follows:

Before redis2.8, slave nodes adopted full synchronization mode after initialization or disconnection reconnection. After 2.8, PSYNC command was introduced to determine whether incremental synchronization was adopted after disconnection reconnection of slave nodes.

3.2 incremental synchronization

PSYNC has full data resynchronization and incremental synchronization modes.

Full-volume resynchronization: It is basically consistent with the old version of replication, which can be understood as "full-volume" replication.

Partial resynchronization: when salve is disconnected and reconnected, in the command propagation stage, only need to send the command executed during this period of disconnection with master to slave, which can be understood as "incremental" replication.

Three concepts are important in PSYNC execution: runid, offset, and replication backlog buffer.

1.runid

Each Redis server has an ID that identifies itself. This ID sent in PSYNC refers to the ID of the previous Master connected. If this ID is not saved, the PSYNC command will use "PSYNC ?" -1"This form is sent to the Master, indicating that full replication is required.

offset (copy offset)

Both Master and Slave copies maintain an offset. Master successfully sends N byte command will increase the offset in Master by N, Slave will also increase the offset in Slave by N after receiving N byte command. Master and Slave should be consistent if their states are consistent.

3. replication backlog buffer

A replication backlog buffer is a fixed-length circular backlog queue (FIFO queue) maintained by the Master to buffer commands that have been propagated. When the Master propagates commands, it not only sends commands to all Slaves, but also writes commands to the replication backlog buffer. The difference between PSYNC execution process and SYNC is that when salve is connected, it is judged whether full synchronization is required. The logic process of full synchronization is the same as SYNC. PSYNC is executed as follows:

When the client sends the SLAVEOF command to the server, that is, when the salve initiates a connection request to the master, the slave determines whether it is the first connection according to whether it saves the Master runid.

If this is the first synchronization, send PSYNC to Master. -1 command for full synchronization; if it is a reconnect, the PSYNC runid offset command is sent to the Master (runid is the identity ID of the master, offset is the global migration amount of the slave synchronization command).

After receiving the PSYNC command, the Master first determines whether the runid is consistent with the ID of the local machine. If it is consistent, it will determine whether the offset offset is different from the offset of the local machine by more than the size of the replication backlog buffer. If not, it will send CONTINUE to the Slave. At this time, the Slave only needs to wait for the Master to return the lost command during the loss of connection. If the runid and native id do not match or the offset difference exceeds the replication backlog buffer size, then FULLRESYNC runid offset is returned, Slave saves the runid and performs full synchronization.

When the master node propagates commands, the master database will transfer each write command to the slave database, and store the write command in the backlog queue, and record the global offset of the command stored in the current backlog queue. When the salve reconnects, the master will find the command executed during this period of time in the circular backlog queue according to the offset transmitted from the node, and synchronize it to the salve node to achieve the incremental synchronization result.

The PSYNC execution process is as follows:

It can be seen from the above execution flow of PSYNC that when the slave node is disconnected and reconnected, the core of determining whether to adopt incremental synchronization is whether the difference between the offset of slave and the offset of master exceeds the size of replication backlog buffer, and this size is configured by the following parameters. Replication backlog buffer is essentially a circular queue of fixed length. By default, the backlog queue size is 1MB. Queue size can be set through configuration file: Set replication backlog buffer size. The larger the backlog queue, the longer the master-slave database is allowed to be disconnected.

repl-backlog-size 1mb

Redis also provides how often the circular queue can be released when there is no slave to synchronize, the default is one hour, and how often the replication backlog buffer can be released when there is no salve connection.

repl-backlog-ttl 3600 IV. Master-slave replication strategy

Redis adopts an optimistic replication strategy, that is, to a certain extent, the contents of the master-slave database are inconsistent, but the final consistency of the master-slave database data is maintained. Specifically, Redis itself is asynchronous in the process of master-slave replication. After the master-slave database executes the client request, it will immediately return the result to the client and synchronize the command to the slave database asynchronously. However, it will not wait for the slave database to be completely synchronized before returning to the client. Although this feature ensures that performance is not affected during master-slave replication, it also produces a time window of data inconsistency. If the network is suddenly disconnected during this time window, it will cause data inconsistency between the two. If you do not add other policies to the configuration file, this is the default. In order to prevent the master-slave inconsistency from being uncontrollable, redis provides the following two parameters to constrain:

min-slaves-to-write 3min-slaves-max-lag 10

When the number of slaves is less than min-slaves-to-write and the latency is less than or equal to min-slaves-max-lag, the master stops writing.

There is another parameter that also affects the delay between master and slave:

repl-disable-tcp-nodelay：

If set to yes, redis will merge small TCP packets to save bandwidth, but will increase synchronization delay, causing master and slave data to be inconsistent. If set to no, redis master will immediately send synchronized data with little delay.

The master-slave synchronization of Redis, regardless of the scenario, can be abstracted into the following seven steps:

1. establishes a socket connection

The slave server creates a socket connection to the master server according to the set socket, and after the master server receives the socket connection from the slave server, it creates a client state for the socket, and regards the slave server as the client of the master server at this time, that is, the slave server has two identities of server and client at the same time.

2. Send PING command

The PING command has two main functions: although the socket connection has been established, it has not been used yet. By sending the PING command, check whether the socket reading and writing status is normal; by sending the PING command, check whether the main server can process the command request normally and can process the main server reply PONG.

3. authentication

The slave server receives a "PONG" reply from the master server, and the next thing to consider is authentication. Authentication occurs if the masterauth option is set from the server, and no authentication occurs if the masterauth option is not set from the server.

4. transmission port information

After the authentication step, the slave will execute the command REPLCONF listening-port, sending the slave's listening port number to the master.

5. data synchronization

The slave server sends SYNC commands and PSYNC commands to the master server to perform synchronization operations.

6. command propagation

The master-slave server enters the command propagation phase, where the master server only needs to send its own write commands to the slave server, and the slave server only needs to execute and receive the write commands sent by the master server.

The above is "Redis master-slave synchronization mechanism example analysis" all the content of this article, thank you for reading! I believe that everyone has a certain understanding, hope to share the content to help everyone, if you still want to learn more knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.