Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to analyze the principle of cluster master-slave replication in Redis. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

This article takes you to deeply understand the principle of master-slave replication of Redis cluster. I hope it will be helpful to you!

First, think about a question: why do you need a distributed solution with such a high performance of redis?

1. Achieve higher performance: for highly concurrent applications, stand-alone performance will be affected, so more redis servers are needed to share the pressure and achieve load balancing.

2. Achieve high availability: if stand-alone, prevent downtime / hardware failure

3. Scalable: single machine memory and hardware are limited to achieve scale-out.

Redundant or fragmented storage implements the above features.

Second, master-slave replication-replication configuration

Like Kafka,Mysql,Rocketmq, redis supports cluster deployment. Cluster nodes are divided into master and slave. The master node is master and the slave node is slave (the latest replica replica). Replication synchronizes the latest data from master through the replication mechanism. Redis provides a very convenient command to enable master-slave replication. [related recommendation: Redis video tutorial]

How to configure to enable master-slave replication?

Take the local pseudo-cluster as an example, port 6379 is the slave node and 6378 is the master node.

1. Configure the replicaof masterip masterport from the node redis.conf after the slave node starts, automatically connect to the master node and start to synchronize data.

Source code description:

/ / execute this method void replicationCron (void) {. / / every 1s to check whether you need to connect to master. If it is in REPL_STATE_CONNECT state, you must connect to master / / # define REPL_STATE_CONNECT 1 Must connect to master if (server.repl_state = = REPL_STATE_CONNECT) {serverLog (LL_NOTICE, "Connecting to MASTER% SVR% d", server.masterhost, server.masterport). / / and master create connection if (connectWithMaster () = = C_OK) {serverLog (LL_NOTICE, "MASTER REPLICA sync started") }} / / send ping command to slave if ((replication_cron_loops% server.repl_ping_slave_period) = = 0 & & listLength (server.slaves)) {/ * Note that we don't send the PING if the clients are paused during * a Redis Cluster manual failover: the PING we send will otherwise * alter the replication offsets of master and slave And will no longer * match the one stored into 'mf_master_offset' state. * / int manual_failover_in_progress = server.cluster_enabled & & server.cluster- > mf_end & & clientsArePaused (); if (! manual_failover_in_progress) {ping_argv [0] = createStringObject ("PING", 4); replicationFeedSlaves (server.slaves, server.slaveseldb, ping_argv, 1) DecrRefCount (ping_argv [0]);}} / / sends a newline character to all slave, telling slave to wait to receive the rdb file listRewind (server.slaves,&li); while ((ln = listNext (& li) {client * slave = ln- > value Int is_presync = (slave- > replstate = = SLAVE_STATE_WAIT_BGSAVE_START | | (slave- > replstate = = SLAVE_STATE_WAIT_BGSAVE_END & & server.rdb_child_type! = RDB_CHILD_TYPE_SOCKET)) If (is_presync) {if (write (slave- > fd, "\ n", 1) =-1) {/ * Don't worry about socket errors, it's just a ping. * /}}.}

3. Full replication process-supports diskless replication or rdb persistent replication

When slave is connected to master, initialize the replication using the psync (formerly sync command, which does not allow partial resynchronization, so now use PSYNC) command to send the primary node replication id and processed maximum offset to master.

The master node has the following two attributes, a replication id (flag instance) and an offset (flag written to the stream of the slave node)

Replication ID, offset

If there is not enough backlog in the primary node buffer, or if the replica references a history that is no longer known (copy ID), a full resynchronization occurs

Source code description:

/ / No rdb process, no aof rewriting process if (server.rdb_child_pid = =-1 & & server.aof_child_pid = =-1) {time_t idle, max_idle = 0; int slaves_waiting = 0; int mincapa =-1; listNode * ln; listIter li; listRewind (server.slaves,&li) While ((ln = listNext (& li)) {client * slave = ln- > value; / / determine whether slave is waiting for bgsave status if (slave- > replstate = = SLAVE_STATE_WAIT_BGSAVE_START) {/ / how long has not sent heartbeat or queried data idle interval idle = server.unixtime-slave- > lastinteraction If (idle > max_idle) max_idle = idle; slaves_waiting++; mincapa = (mincapa =-1)? Slave- > slave_capa: (mincapa & slave- > slave_capa);}} if (slaves_waiting & & (! server.repl_diskless_sync | | max_idle > server.repl_diskless_sync_delay) {/ * Start the BGSAVE. The called function may start a * BGSAVE with socket target or disk target depending on the * configuration and slaves capabilities. * / bgsave rdb generates startBgsaveForReplication (mincapa);}}

Slave state transition process during replication.

IV. Copy id understanding

Each time the instance is restarted from scratch as the primary instance, or the replica is promoted to the primary instance, a new replication ID is generated for that instance. If the replication id of two replica is the same, they may have the same data at different times, and the offset is understood as a logical time for a given history of keeping the latest dataset (replication ID). It needs to be judged by Replication ID and offset data. It is used to determine where to synchronize data from the node.

Fifth, frequently asked questions about master-slave replication

1. What happens if slave has its own data?

Slave deletes its own data before loading it with a rdb file.

2. How to deal with client write commands in the process of generating rdb files?

Save it to the in-memory cache, and send it to slave after the rdb is sent.

3. How does Redis replication handle key expiration?

1. The replica does not expire the key, but waits for the host to expire the key. When the host expires key (or expels it because of LRU), it synthesizes a DEL command that is transferred to all copies.

2. However, due to the host-driven expire, sometimes the copy may still have a logically expired memory key because the primary server cannot provide the DEL command in time. To address this problem, the replica uses its logical clock to report that a key does not exist and is only used for read operations that do not violate dataset consistency (because new commands from the primary server will arrive)

3. Key expiration is not performed during Lua script execution. When the Lua script runs, conceptually, the time in the primary node is frozen, so the given key will exist or not for all the time the script runs. This prevents key from expiring in the middle of the script, and requires key to send the same script to the copy in a manner that ensures the same effect in the dataset.

Once the replica is promoted to the primary replica, it starts to expire the key independently and does not need any help from the old master replica.

VI. Master-slave replication summary

1. The problem of data backup is solved, but the rdb file is large, the transfer is large, and the recovery time is also long.

2. If the master is abnormal, you need to manually elect replica to master.

3. In the case of 1 master and multi-slave, 1 master and 1 slave, there is still a single point problem.

4. Redis version 2.8.18 supports diskless replication with higher performance.

VII. Copy instructions

1. Asynchronous replication is used by default, and the number of synchronous commands is confirmed asynchronously.

2. 1 master can have multiple copies

3. Replicas can also have their own replicas. Starting from redis4.0, replicas will receive exactly the same replication stream from the primary node.

4. Replication can be used either for extensibility or for multiple copies of a read-only query

On how to analyze the Redis cluster master from the replication principle to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.