Shulou(Shulou.com)05/31 Report--

This article mainly introduces the example analysis of master-slave replication architecture in Redis, which is very detailed and has certain reference value. Friends who are interested must finish it!

High availability has two meanings: one is not to lose data as much as possible, and the other is to provide services as much as possible. AOF and RDB ensure that data persistence is not lost as much as possible, while master-slave replication is to add copies, and one piece of data is saved to multiple instances. Even if one instance goes down, other instances can still provide services.

The main purpose of this article is to fully understand one of the Redis high availability technology solutions, master-slave replication architecture.

This article hard core, it is suggested that the collection slowly taste, I believe readers will have a qualitative improvement. If there are any mistakes, please correct them. Thank you.

Core knowledge points

Start by sending a message

Question = opportunity. When you encounter a problem, you are actually happy in your heart. the bigger the problem, the greater the opportunity.

Everything has a price, there is a gain, there is a loss, there is a gain, so we don't have to worry about a lot of things, we just need to figure out what we want to do and what price we are willing to pay for it, and then let it go!

1. Master-Slave replication Overview 65 Brother: with RDB and AOF, we are no longer afraid of downtime and loss of data, but how to achieve high availability when Redis instances are down?

Since one machine is down and unable to provide service, what about many? Is it possible to solve it? Redis provides a master-slave mode in which a redundant copy of the data is replicated to another Redis server through master-slave replication.

The former is called master node (master) and the latter is called slave node (slave); data replication is one-way and can only be from master node to slave node.

By default, each Redis server is a master node; and a master node can have multiple slave nodes (or no slave nodes), but a slave node can have only one master node.

65 Brother: how to ensure the consistency of the data between master and slave?

In order to ensure the consistency of the replica data, the master-slave architecture adopts the method of separation of read and write.

Read operation: both master and slave libraries can be executed

Write operation: the master library executes first, and then synchronizes the write operation to the slave library

65 Brother: why should we adopt the method of separation of reading and writing?

We can assume that both master and slave libraries can execute write instructions. if the same data is modified many times, and each change is sent to a different master-slave instance, the copy data of the instance will be inconsistent.

If Redis needs to lock and coordinate the modification of multiple instances in order to ensure data consistency, Redis will not do so!

65 Brother: is there any other function of master-slave replication?

Fault recovery: when the primary node is down, other nodes can still provide services

Load balancing: Master nodes provide write services, and Slave nodes provide read services to share the pressure.

High availability Cornerstone: is the foundation of Sentinel and cluster implementation, is the high availability cornerstone.

two。 Build master-slave replication

The start of master-slave replication is completely initiated by the slave node, and we don't need to do anything on the master node.

65 Brother: how to build a master-slave replication architecture?

The relationship between the master library and the slave library can be formed through the replicaof command (slaveof was used before Redis 5.0).

There are three ways to enable master-slave replication in the slave node:

Configuration file

Add replicaof to the configuration file of the slave server

Start command

Add-- replicaof after the redis-server startup command

Client command

After starting multiple Redis instances, execute the command directly through the client: replicaof, then the Redis instance becomes a slave node.

For example, suppose you now have instance 1 (172.16.88.1), instance 2 (172.16.88.2), and instance 3 (172.16.88.3). If you execute the following commands on instance 2 and instance 3, respectively, instance 2 and instance 3 become the slave library of instance 1, and instance 1 becomes Master.

Replicaof 172.16.88.1 63793. Principle of master-slave replication

Once the master-slave library mode adopts read-write separation, all data writing operations will only be carried out on the master database, and there is no need to coordinate three instances.

After the master database has the latest data, it will be synchronized to the slave database, so that the data of the master-slave database is consistent.

65 Brother: how is the synchronization of master and slave libraries completed? Is the master database data transferred to the slave database at once, or is it synchronized in batches? How to synchronize in normal operation? If the network between the master and slave libraries is disconnected, can the data remain consistent after reconnecting?

65 Brother, why do you have so many questions? synchronization is divided into three situations:

The first full copy of the master-slave library

Synchronization during normal operation of master and slave

Disconnect and reconnect synchronization between master and slave libraries.

Master-slave library first full copy 65 elder brother: I am so dizzy, let's start with the first synchronization between master and slave libraries.

The first replication process of the master-slave library can be divided into three stages: the connection establishment stage (that is, the preparation stage), the master library synchronization data to the slave library stage, and the sending of new write commands during synchronization to the slave library stage.

Directly above the picture, there is a perception of the overall view as a whole, which is described in detail later.

Establish a connection

The main function of this stage is to establish a connection between master and slave nodes to prepare for full data synchronization. The slave library will establish a connection with the master library, execute replicaof and send psync commands to the slave library and tell the master library that synchronization is imminent. After the master library confirms the reply, the master and slave libraries begin to synchronize.

65 Brother: how does the slave library know the main database information and establish a connection?

After configuring the IP and port of the master node in the replicaof configuration item in the slave node's configuration file, the slave node knows that it is going to connect to that master node.

Two fields, masterhost and masterport, are maintained from inside the node to store the IP and port information of the master node.

The slave library executes replicaof and sends the psync command, indicating that the data synchronization is to be performed. After receiving the command, the master library starts the replication according to the parameters. The command contains two parameters: the runID of the main library and the offset of the replication progress.

RunID: a unique identity ID is automatically generated when each Redis instance is started. For the first master-slave replication, the master library runID is not known, and the parameter is set to "?".

Offset: the first replication is set to-1, which indicates the first replication and records the replication progress offset.

After the master library receives the psync command, it responds to the command with FULLRESYNC with two parameters: the master library runID and the master library's current replication progress offset, which are returned to the slave library. When a response is received from the library, these two parameters are recorded.

The FULLRESYNC response indicates the full replication used for the first replication, that is, the master database copies all the current data to the slave database.

The master library synchronizes data to the slave database

The second stage

Master executes the bgsave command to generate the RDB file and sends the file to the slave library, while the master library opens up a replication buffer buffer for each slave to record all write commands received since the generation of the RDB file.

After receiving the RDB file from the library, save it to disk, empty the data in the current database, and then load the RDB file data into memory.

Send a new write command to the slave library

The third stage

After loading the RDB from the slave node, the master node sends the data of the replication buffer buffer to the slave node, the Slave receives and executes, and the slave node synchronizes to the same state of the master node.

65 Brother: can the request be accepted normally when the master database synchronizes the data to the slave database?

The main library will not be blocked, Redis, as the only fast man, how can easily block it.

The write operation after the generation of the RDB file is not recorded in the RDB file just now, in order to ensure the consistency of the master and slave library data, the main library will use a replication buffer in memory to record all write operations after the generation of the RDB file.

65 Brother: why do you want to empty the current database after receiving the RDB file from the library?

Because the slave library may have saved other data before synchronizing with the master library through the replcaof command to prevent the influence between master and slave data.

What on earth is replication buffer?

A buffer created on the master side holds all the master data writes in the following three times.

1) master performs write operations during the period when bgsave generates RDB

2) write operation during master sending rdb to slave network transmission

3) the write operation during which the slave load rdb file restores the data to memory.

Whether Redis communicates with the client or communicates with the slave library, Redis allocates a memory buffer for data exchange. The client is a client and the slave library is also a client. After each client is connected to the Redis, the Redis allocates a proprietary client buffer, and all data interactions are carried out through this buffer.

Master first writes the data to this buffer and then sends it out over the network, thus completing the data exchange.

Master assigns a buffer to the master-slave whether it is in incremental synchronization or full synchronization, but this buffer is specifically used to propagate write commands to the slave library to ensure that the master-slave data is consistent. We usually call it replication buffer.

Problems caused by too small replication buffer:

Replication buffer is set by client-output-buffer-limit slave, and when this value is too small, the master-slave replication connection will be disconnected.

1) when the master-slave replication connection is disconnected, master releases the data related to the connection. The data in the replication buffer is also lost, and the replication process is resumed between the master and slave.

2) there is a more serious problem, in which the master-slave replication connection is disconnected, resulting in an infinite cycle of reperforming bgsave and rdb retransmission operations on the master-slave.

When the amount of data of the master node is large, or the network delay between the master node and the slave node is large, the size of the buffer may exceed the limit, and the master node will disconnect from the slave node.

This situation may cause full copy-> replication buffer overflow causes connection break-> reconnect-> full copy-> replication buffer buffer overflow causes connection break. The cycle.

Details: [top redis headaches for devops-replication buffer]

Therefore, it is recommended that the hard/soft limit of replication buffer be set to 512m.

Config set client-output-buffer-limit "slave 536870912 536870912 0" 65 Brother: why not use AOF for master-slave library replication? Less data is lost than RDB.

This is a good question for the following reasons:

RDB files are binary files, and the IO efficiency of network transferring RDB and writing to disk is higher than that of AOF.

RDB is also more efficient than AOF when recovering data from the library.

Incremental copy 65 Brother: what if the network between the master and slave libraries is cut off? Do you want to copy all of it again after disconnection?

Before Redis 2.8.If the master and slave library had a network flash when the command was propagated, the slave library would make a full copy with the master library again, which was very expensive.

Starting from Redis 2.8.After the network is down, the master-slave library will continue to synchronize by means of incremental replication.

Incremental replication: for replication after a network interruption, only the write commands executed by the master node during the interruption are sent to the slave node, which is more efficient than full replication.

Repl_backlog_buffer

The secret of disconnecting and reconnecting incremental copy is the repl_backlog_buffer buffer. No matter when master records write instructions in repl_backlog_buffer, because memory is limited, repl_backlog_buffer is a fixed-length circular array. If the array is full, it will overwrite the previous content from scratch.

Master uses master_repl_offset to record the position offset it writes to, and slave uses slave_repl_offset to record the offset it has read.

When the master receives a write operation, the offset increases. As synchronous write instructions continue to be executed from the library, the replicated offset slave_repl_offset in repl_backlog_buffer is also increasing.

Normally, the two offsets are basically the same. During the network disconnection phase, the main library may receive new write commands, so master_repl_offset will be greater than slave_repl_offset.

When the master and slave are disconnected and reconnected, slave first sends a psync command to master and sends its own runID,slave_repl_offset to master.

Master only needs to synchronize the commands between master_repl_offset and slave_repl_offset to the slave library.

The incremental copy execution process is shown below:

65 Brother: if the repl_backlog_buffer is too small, it will be overwritten by the new write operation of Master before it can be read from the library.

We need to find a way to avoid this situation, and once it is overwritten, we will perform a full copy. We can adjust the repl_backlog_size parameter to control the buffer size. Calculation formula:

Repl_backlog_buffer = second * write_size_per_second

Second: the average time it takes to disconnect and reconnect the primary server from the server

Average amount of command data generated by write_size_per_second:master per second (sum of write commands and data sizes)

For example, if the master server produces an average of 1 MB of write data per second, and it takes an average of 5 seconds to reconnect to the master server after being disconnected from the slave server, the size of the replication backlog buffer cannot be less than 5 MB.

For security reasons, you can set the size of the replication backlog buffer to 2 * second * write_size_per_second, which ensures that most cases of disconnection can be handled with partial resynchronization.

Command propagation based on long connection 65 Brother: after completing the full synchronization, how to synchronize the normal operation process?

When the master and slave libraries complete full replication, a network connection is maintained between them, and the master library synchronizes subsequent command operations to the slave database through this connection. This process is also known as command propagation based on persistent connections. The purpose of using persistent connections is to avoid the overhead caused by frequent connections.

In the command propagation phase, in addition to sending write commands, the master and slave nodes also maintain the heartbeat mechanism: PING and REPLCONF ACK.

Master-> slave: PING

Every specified time, the master node will send a PING command to the slave node. The main purpose of this PING command is to let the slave node make a timeout judgment.

Slave-> Master: REPLCONF ACK

During the command propagation phase, the slave server sends commands to the master server at a frequency of once per second by default:

REPLCONF ACK

Where replication_offset is the current replication offset from the server. Sending REPLCONF ACK commands has three functions for master and slave servers:

Detect the network connection status of the master-slave server.

Auxiliary implementation of the min-slaves option.

The detection command is missing, and the slave node sends its own slave_replication_offset, and the master node will compare with its own master_replication_offset. If the slave node data is missing, the master node will find and push the missing data from the repl_backlog_buffer buffer. Note that the offset and repl_backlog_buffer buffers can be used not only for partial replication, but also for situations such as command loss; the difference is that the former is done after disconnection reconnection, while the latter is done when the master and slave nodes are not disconnected.

How to determine whether to perform full synchronization or partial synchronization?

In Redis 2.8and later, slave nodes can send psync commands to request synchronization of data. At this time, depending on the current state of master and slave nodes, the synchronization mode may be full replication or partial replication. This article takes Redis 2.8and later as an example.

The key is the implementation of psync:

According to the current state, the slave node sends a psync command to master:

If the slave node has never performed replicaof, then send psync?-1 from the slave node and send a full copy request to the master node

Send psync if the slave node has previously performed replicaof, and runID is the master node saved by the last replication runID,offset is the replication offset saved by the slave node as of the last replication.

The master node decides whether to perform a full or partial replication based on the psync command received and the current server status:

RunID is the same as the runID sent by the slave node, and the data after the slave_repl_offset sent by the slave node exists in the repl_backlog_buffer buffer, then reply CONTINUE, indicating that partial replication will be carried out, and the slave node can wait for the master node to send its missing data.

If the runID is different from the runID sent by the slave node, or the data after the slave_repl_offset sent by the slave node is no longer in the repl_backlog_buffer buffer of the master node (squeezed out in the queue), reply to the slave node FULLRESYNC, indicating that you want to make a full copy, where runID indicates that the current runID,offset of the master node represents the current offset of the master node, and the slave node saves these two values for use.

If a slave database is disconnected from the master database for too long, its data in the slave_repl_offset location of the master library repl_backlog_buffer has been overwritten, and a full copy will be made between the slave database and the master database.

To sum up

Each slave library records its own slave_repl_offset, and the replication progress of each slave library is not necessarily the same.

When reconnecting with the master database for recovery, the slave database will send its recorded slave_repl_offset to the master database through the psync command, and the master database will decide whether the slave database can be copied incrementally or in full according to the replication progress of the slave database.

Replication buffer and repl_backlog

Replication buffer corresponds to each slave and is set through config set client-output-buffer-limit slave.

Repl_backlog_buffer is a ring buffer, there will be only one in the whole master process, and all slave are common. The size of repl_backlog is set by the repl-backlog-size parameter, and the default size is 1m. The size of the backlog buffer can be estimated based on the sum of commands generated per second, (master executes rdb bgsave) + (master sends rdb to slave) + (slave load rdb file), and the repl-backlog-size value is not less than the product of the two.

Generally speaking, replication buffer is the buffer on the master library for the clients connected to the slave library when the master and slave libraries make a full copy, while repl_backlog_buffer is a dedicated buffer on the master library for continuous write operations to support incremental replication from the slave library.

Repl_backlog_buffer is a dedicated buffer that starts to receive write commands after the Redis server is started, which is shared by all slave libraries. The master library and the slave library record their own replication progress, so different slave libraries will send their own replication progress (slave_repl_offset) to the master library during recovery, so that the master library can synchronize with it independently.

As shown in the figure:

4. Master-slave application problem 4.1 separation of reading and writing

Data expiration problem

65 Brother: in the master-slave replication scenario, will the slave node delete expired data?

This is a good question. For the sake of data consistency between master and slave nodes, slave nodes will not actively delete data. We know that Redis has two deletion strategies:

Lazy deletion: when the client queries the corresponding data, Redis determines whether the data is out of date, and then deletes it.

Delete periodically: Redis deletes expired data through scheduled tasks.

65 Brother: will the client read the expired data by reading the data from the node?

Redis 3.2 begins to determine whether the data is out of date by reading data from the node. If it expires, the client is not returned, and the data is deleted.

4.2 stand-alone memory size limit

If the Redis stand-alone memory reaches 10GB, the synchronization time of a slave node is at the level of a few minutes; if there are more slave nodes, the recovery speed will be slower. If the read load of the system is very high, and the slave node can not provide services during this period of time, it will cause a lot of pressure on the system.

If the amount of data is too large, it takes too long for the master node to save RDB files in the full replication phase, and the slave node cannot receive data for a long time to trigger a timeout. The data synchronization of the master and slave nodes may also fall into full replication-> timeout causes replication interruption-> reconnection-> full replication-> timeout leads to replication interruption. The cycle.

In addition, except for the absolute amount of stand-alone memory of the master node, the proportion of the host memory occupied by it should not be too large: it is best to use only 50%-65% of the memory, leaving 30% 45% of the memory for executing bgsave commands and creating replication buffers.

Summary

The role of master-slave replication: AOF and RDB binaries ensure fast data recovery during downtime and prevent data loss as much as possible. However, the service is still unable to provide after the downtime, so the master-slave architecture and the separation of read and write have evolved.

Principle of master-slave replication: connection establishment stage, data synchronization stage, command propagation stage; data synchronization stage is divided into full replication and partial replication; command propagation phase, there are PING and REPLCONF ACK commands between master and slave nodes to detect the heartbeat of each other.

Although master-slave replication solves or alleviates the problems of data redundancy, fault recovery and read load balancing, its shortcomings are still obvious: failure recovery cannot be automated; write operations cannot be load balanced; storage capacity is limited by a single machine; to solve these problems, we need the help of sentinels and clusters, which I will introduce in later articles. Welcome to follow.

65 Brother: brother Ma, your pictures are really beautiful and the content is good. I have gained a lot from your article. I want to collect, like, read and share. Let more outstanding developers see common progress! The above is all the contents of the article "sample Analysis of Master-Slave replication Architecture in Redis". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.