Shulou(Shulou.com)06/01 Report--

Today I would like to share with you about Redis master-slave synchronization (also known as "replication").

We know that when there are multiple Redis servers, there must be one master server and multiple slave servers. Generally speaking, the master server writes and reads from the server.

So there is a question: how does the slave server synchronize with the master server?

This problem is solved through today's content: master-slave synchronization.

The content of the article is still relatively dry. I suggest you calm down and concentrate on it. I will give you a brief summary at the end of the article.

1. How to synchronize master and slave

Suppose there are two Redis servers with addresses of 127.0.0.1 and 127.0.0.1, respectively.

We enter the command on the client side of 127.0.0.1purl 12345:

127.0.0.1 SLAVEOF 12345 > 127.0.0.6379

Thus, the 127.0.0.1 12345 server will replicate the data of 127.0.0.1 VR 6379. That is, the former is the slave server and the latter is the master server.

In addition to copying in the above way, you can also set it through the slaveof option in the configuration file.

Perhaps, curious you will want to know, Redis is how to achieve master-slave synchronization?

Ok, let's take a closer look.

two。 The realization process of master-slave synchronization

Master-slave synchronization is divided into two steps: synchronization and command propagation.

Synchronization: updates the database state of the slave server to the current database state of the master server. (what does the database state mean mentioned at the beginning of this article? for those who are not clear, you can take a look at "persistence" first.) Command propagation: when the state of the master server database is modified, the state of the master server database is inconsistent. At this point, it is necessary to synchronize the master-slave data to the consistent process.

The above is the role of master-slave synchronization of the two steps, now I intend to elaborate on the implementation process of these two steps.

It is important to note in advance that before Redis 2.8, the above two steps must be performed sequentially during master-slave replication, while from 2.8 onwards, you may only need to execute command propagation. Why is this explained below?

2.1 synchronization

The synchronization operation from the slave server to the master service needs to be realized through the sync command. The following are the steps to execute the sync command:

After the sync command is sent from the server to the master server, the master server executes the bgsave command, which is used to generate the rdb file, and records the write commands executed from now on in a buffer. After the bgsave execution is completed, the generated rdb file is sent to the slave server to update the data to the slave server, and then the write command of the buffer record is sent to the slave server. After the slave server executes these write commands, the database state is consistent with that of the master server.

This is what it looks like in a graph:

2.2 Command propagation

After the synchronization operation, the master-slave database state is actually consistent at this time, but this consistent state is not immutable.

After the synchronization is completed, the master server may immediately receive a new write command, and after executing the command, the state of the master-slave database is inconsistent.

In order to make the state of the master-slave database consistent again, the master server needs to propagate the command to the slave server, that is, send the write command that has just caused inconsistency to the slave server for execution. After the execution of the slave server is complete, the state of the master-slave database is consistent again.

Here's a question:

I wonder if there are readers who think that when the above inconsistencies occur, won't it be ok for Redis to synchronize again?

It is true that synchronization can be restored in terms of effect, but it is not necessary. The reason is that the sync command to achieve synchronization is a very resource-consuming operation. After reading the description in the following figure, I am sure you will understand it.

Since synchronization is a very resource-consuming operation, is there any optimization method for Redis? Of course the answer is yes.

2.3 optimized synchronous operation

Remember what I said above-starting with version 2.8, master-slave synchronization may only need to execute command propagation. This is also because sync consumes more resources, so it takes optimization.

So when can we do this? Let's take a look at the prerequisites:

Master-slave synchronization is actually divided into two situations:

Initial replication: replicate from the server for the first time after the current master server (PS: the master server is possible to be replaced) is disconnected: the master-slave server in the command propagation stage interrupts replication due to network problems, and the slave server reconnects to the master server and continues replication through automatic reconnection.

In the case of re-replication after disconnection, synchronization (sync command) and command propagation will be performed again before version 2.8.

It is also very inefficient to say that during the disconnection, the master server (which has tens of thousands of key-value pairs) has only executed a few write commands, but in order for the slave server to make up for these commands, it is also very inefficient to re-execute sync to generate a new rdb file.

To solve this problem, 2.8 started using the psync command instead of the sync command to perform synchronization operations.

Psync has two modes: full resynchronization and partial resynchronization:

Full resynchronization: for initial replication, the execution process is the same as sync, so I won't repeat it here. Partial resynchronization: used for repetition after disconnection. If certain conditions are met, the master server only needs to send the write commands executed during the disconnection to the slave server.

Therefore, it is obvious that when the master-slave synchronization is disconnected and replicated, the partial resynchronization mode of psync can solve the inefficiency of sync.

In the above introduction, there is a "meet certain conditions", what are the conditions for ghosts? -- in fact, it is a comparison of offsets, which can be read on.

2.4 implementation of partial resynchronization

The partial resynchronization function consists of the following three parts:

Replication offset of master-slave server replication backlog buffer server of master server running id (run id)

2.4.1 replication offset

The master and slave servers that perform the replication maintain their own replication offsets:

Each time the master server propagates n bytes of data to the slave server, it adds n to its own replication offset. When receiving data from the master server from the server, it will also add its own replication offset to n.

For example:

If the replication offset of the current master server is 10000, 30 bytes of data are propagated to the slave server at this time, and the replication offset is 10030 at the end.

At this time, the slave server is disconnected before receiving the 30 bytes of data, and after reconnecting, the replication offset of the slave server is still 10000, indicating that the master-slave data is inconsistent, and a psync command will be sent to the master server.

So should the master server perform a full resynchronization or partial resynchronization of the slave server? If partial resynchronization is performed, how does the master server know which data to synchronize to the slave server?

The following answers are all related to copying the backlog of buffers.

2.4.2 copy backlog buffer

First of all, the copy backlog buffer is a fixed-length, first-in, first-out queue, default 1MB.

When the master server propagates the command, the command is sent not only to the slave server, but also to the buffer.

So the structure of the replication backlog buffer is as follows:

When sending psync commands from the server to the master server, you also need to bring your own replication offset, and the master server can compare this replication offset with the offset of the replication backlog buffer.

If there is data after the replication offset of + 1 from the server in the replication backlog buffer, a partial resynchronization will be performed, otherwise a full resynchronization will be performed.

2.4.3 run id

When you run id for the first time, the master server will send its own running id to the slave server to save it.

When the slave server is disconnected and reconnected, the slave server sends the running id to the master server that just connected.

If the id of the current server is the same, it means that the server replicated before the slave server is disconnected is the current server, and the master server can try to perform partial synchronization; if different, the server copied before the slave server is disconnected is not the current server, and the master server directly performs full resynchronization.

Spent a lot of ink, and finally finished the implementation of partial resynchronization, and finally added an auxiliary function.

2.5 heartbeat detection

As mentioned just now, master-slave synchronization has two steps: synchronization and command propagation.

When the synchronization is completed, the master-slave server enters the command propagation phase, and the slave server sends a command to the master server at a frequency of once per second: REPLCONF ACK, where replication_offset is the current replication offset of the slave server.

Sending this command has three main functions:

Detect the network status of the master-slave server to assist the implementation of the min-slaves option to detect the loss of commands (if lost, the master server will re-send the lost write commands to the slave server)

3. Summary

Finally finished writing the final content, a few hours have passed, let's sum up the content of this article:

Send SLAVEOF command for master-slave synchronization, for example: SLAVEOF 127.0.0.6379

Master-slave synchronization has two steps: synchronization and command propagation.

Synchronization: the command to update the database state of the slave server to the current database state of the master server (a resource-consuming operation) propagates: when the database state of the master server is modified, the database state of the master server is inconsistent. At this point, it is necessary to synchronize the master-slave data to be consistent.

Master-slave synchronization can be divided into two cases: the first copy and the repeated system after disconnection.

Starting from version 2.8, in the event of a repeat after a disconnection, the primary server will determine whether to perform a full or partial resynchronization based on the replication offset, replication backlog buffer and run id.

Version 2.8 uses the psync command instead of the sync command to perform synchronization operations. The purpose is to solve the inefficient operation of synchronization (sync command)

The above is the whole content of this article. I hope the content of this article has a certain reference and learning value for everyone's study or work. Thank you for your support. If you want to know more about it, please see the relevant links below.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.