Shulou(Shulou.com)05/31 Report--

This article mainly introduces the relevant knowledge of "what is a Redis high-availability cluster". The editor shows you the operation process through an actual case, and the operation method is simple, fast and practical. I hope this article "what is a Redis high-availability cluster" can help you solve the problem.

Several solutions for Redis high availability. Including: "master-slave mode", "sentry mechanism" and "sentry cluster".

The "master-slave mode" has the advantages of separation of reading and writing, sharing of reading pressure, data backup, providing multiple copies and so on.

The Sentinel mechanism can automatically promote the slave node to the master node after the master node failure, and the service availability can be restored without human intervention.

"Sentinel Cluster" solves the problem of single point failure and "misjudgment" caused by single Sentinel.

Redis from the simplest stand-alone version, through data persistence, master-slave multiple copies, Sentinel cluster, through such optimization, both performance and stability are getting higher and higher.

However, with the development of time, the volume of the company's business has ushered in explosive growth, at this time the architecture model, can still bear such a large amount of traffic?

For example, there is such a requirement: to save 50 million key-value pairs with Redis, each key-value pair is about 512B. In order to quickly deploy and provide services, we use the CVM to run the Redis instance. Then, how to select the memory capacity of the CVM?

By calculation, the memory space occupied by these key-value pairs is about 25GB (50 million * 512B).

The first solution that comes to mind is to choose a cloud host with 32GB memory to deploy Redis. Because the memory of 32GB can hold all data, and there is still 7GB, it can ensure the normal operation of the system.

At the same time, RDB is also used to persist the data to ensure that the data can be recovered from the RDB after the failure of the Redis instance.

However, in the course of use, you will find that the response of Redis can sometimes be very slow. Looking at the latest_fork_usec indicator value of Redis (which represents the time spent on the most recent fork) through the INFO command, it turns out that this indicator value is particularly high.

This has something to do with the persistence mechanism of Redis.

When using RDB for persistence, Redis will fork the child process to complete, the time of fork operation is positively related to the amount of data of Redis, and fork will block the main thread during execution. The larger the amount of data, the longer the main thread blocks caused by the fork operation.

Therefore, when using RDB to persist the data of 25GB, the amount of data is large, and the child processes running in the background block the main thread when the fork is created, which causes the Redis response to slow down.

Obviously, this plan is not feasible, we have to look for other options.

How do I save more data?

In order to preserve a large amount of data, we generally have two methods: "scale-up" and "scale-out":

Scale-up: upgrade the resource configuration of a single Redis instance, including increasing memory capacity, increasing disk capacity, and using a higher configuration of CPU

Scale-out: horizontally increase the number of current Redis instances.

First of all, the advantage of "vertical expansion" is that it is simple and direct to implement. However, the plan also faces two potential problems.

The first problem is that when using RDB to persist data, if the amount of data increases, the memory required will also increase, and the main thread may block when the fork child process.

The second problem: vertical expansion will be limited by hardware and cost. This is easy to understand. After all, it is easy to extend memory from 32GB to 64GB, but if you want to expand to 1TB, you will face hardware capacity and cost constraints.

Compared with "vertical expansion", "scale-out" is a more scalable solution. This is because if you want to save more data, you only need to increase the number of instances of Redis without worrying about the hardware and cost limitations of a single instance.

Redis cluster is based on "scale-out". It forms a cluster by starting multiple Redis instances, and then divides the received data into multiple parts according to certain rules, each of which is saved by an instance.

Redis cluster

Redis cluster is a distributed database scheme, which shares data through sharding (also known as slicing) and provides replication and failover functions.

Going back to the scenario we just saw, if you divide the 25GB data into 5 parts on average (of course, you can not divide it equally), use 5 instances to save it, and each instance only needs to save the 5GB data. As shown in the following figure:

Then, in a slicing cluster, when an instance generates RDB for 5GB data, the amount of data is much smaller, and the fork child process generally does not block the main thread for a long time.

After using multiple instances to save data slices, we can not only save the 25GB data, but also avoid the sudden slow response caused by the fork child process blocking the main thread.

In the practical application of Redis, with the expansion of the scale of the business, it is usually inevitable to save a large amount of data. And Redis cluster is a very good solution.

Let's start to study how to build a Redis cluster.

Set up Redis cluster

A Redis cluster is usually composed of multiple nodes. At the beginning, each node is independent of each other, and there is no correlation between the nodes. To build a working cluster, we must connect individual nodes to form a cluster containing multiple nodes.

We can connect the nodes with the CLUSTER MEET command:

CLUSTER MEET

Ip: ip of the node to join the cluster

Port: port of the node to join the cluster

Command description: by sending a CLUSTER MEET command to one node A, you can have the node A receiving the command add another node B to the cluster where node An is located.

This is a bit abstract. Let's take a look at an example.

Suppose you now have three separate nodes, 127.0.0.1, 7001, 127.0.0.1, and 127.0.0.1, which are 7003.

We first use the client to connect to node 7001:

$redis-cli-c-p 7001

Then send a command to node 7001 to add node 7002 to the cluster where 7001 is located:

127.0.0.1 CLUSTER MEET 7001 > 127.0.0.1 7002

Similarly, we send commands to 7003 and add them to the clusters where 7001 and 7002 are located.

127.0.0.1 CLUSTER MEET 7001 > 127.0.0.1 7003

You can view the node information in the cluster through the CLUSTER NODES command.

The cluster now contains three nodes: 7001, 7002, and 7003. However, when using a single instance, it is very clear where the data is stored and where the client accesses it. However, slicing cluster inevitably involves the distributed management of multiple instances.

To use slicing clusters, we need to solve two major problems:

After the data is sliced, how is it distributed among multiple instances?

How does the client determine on which instance the data it wants to access is on?

Next, we will solve the problem one by one.

The corresponding distribution relationship between data slices and instances

In a slicing cluster, the data needs to be distributed on different instances, so how do the data correspond to the instances?

This is related to the Redis Cluster scheme that we are going to talk about next. However, we need to understand the connection and difference between slicing cluster and Redis Cluster first.

Before Redis 3.0, officials did not provide a specific solution for slicing clusters. Since 3. 0, officials have provided a solution called Redis Cluster for implementing slicing clusters.

In fact, slicing cluster is a general mechanism for storing large amounts of data, and this mechanism can be implemented in different ways. The corresponding rules for data and instances are specified in the Redis Cluster scheme.

Specifically, the Redis Cluster scheme uses hash slots (Hash Slot) to handle the mapping between data and instances.

Mapping between Hash slot and Redis instance

In the Redis Cluster scenario, a slice cluster has 16384 hash slots (2 ^ 14). These hash slots are similar to data partitions, and each key-value pair is mapped to a hash slot according to its key.

In the above analysis, 7001, 7002, and 7003 nodes are connected to the same cluster through the CLUSTER MEET command, but the cluster is currently offline because the three nodes in the cluster are not assigned any slots.

So how are these hash slots mapped to specific Redis instances?

We can use the CLUSTER MEET command to manually establish connections between instances to form a cluster, and then use the CLUSTER ADDSLOTS command to specify the number of hash slots on each instance.

CLUSTER ADDSLOTS [slot...]

Redis5.0 provides the CLUSTER CREATE command to create a cluster, and with this command, Redis automatically distributes these slots evenly over the cluster instance.

For example, we assign slots to 7001, 7002, and 7003 nodes with the following command.

Assign slot 0 to slot 5000 to 7001:

127.0.0.1 CLUSTER ADDSLOTS 7001 > 01 2 3 4... 5000

Assign slot 5001 to slot 10000 to 7002:

127.0.0.1 CLUSTER ADDSLOTS 7002 > 5002 5003 5004... 10000

Assign slot 10001to slot 16383 to 7003:

127.0.0.1 CLUSTER ADDSLOTS 7003 > 10002 10003 10004... 16383

When all three CLUSTER ADDSLOTS commands are executed, 16384 slots in the database have been assigned to the corresponding nodes, and the cluster goes online.

Through the hash slot, the slice cluster realizes the distribution of data to the hash slot, the hash slot and then to the instance.

However, even if the instance has the mapping information of the hash slot, how does the client know which instance the data to access is on?

How does the client locate the data?

Generally speaking, after the client and the cluster instance establish a connection, the instance will send the allocation information of the hash slot to the client. However, when the cluster was first created, each instance only knew which hash slots were assigned to it, and did not know the hash slot information that other instances had.

So how can the client get all the hash slot information when it accesses any instance?

The Redis instance will send its own hash slot information to other instances connected to it to complete the diffusion of hash slot allocation information. When the instances are connected to each other, each instance has the mapping relationship of all hash slots.

When the client receives the hash slot information, it caches the hash slot information locally. When the client requests a key-value pair, the hash slot corresponding to the key is calculated first, and then the request can be sent to the corresponding instance.

When the client requests a key-value pair from the node, the node receiving the command calculates which slot the database key to be processed by the command belongs to and checks whether the slot is assigned to it:

If the slot in which the key is located happens to be assigned to the current node, the node will execute the command directly

If it is not assigned to the current node, the node returns a MOVED error to the client, redirect it to the correct node, and sends the command to be executed again.

Which slot does the key belong to?

The node defines which slot the key belongs to by the following algorithm:

Crc16 (key,keylen) & 0x3FFF

Crc16: CRC-16 checksum used to calculate key

0x3FFF: it's 16383 in decimal.

& 0x3FFF: used to calculate an integer between 0,16383 as the slot number of key.

You can see which slot key belongs to through the CLUSTER KEYSLOT command.

Determine whether the slot is handled by the current node

When the node calculates the slot I to which the key belongs, the node determines whether the slot I is assigned to it. So how to judge?

Each node maintains a "slots array", and the node checks slots [I] to determine whether slot I is its own responsibility:

If the corresponding node of slots [I] is the current node, then the current node is responsible for slot I, and the node can execute the commands sent by the client.

If slots [I] does not correspond to the current node, the node will return a MOVED error to the client based on the node pointed to by slots [I], directing the client to the correct node.

MOVED error

Format:

MOVED:

Slot: the slot where the key is located

Ip: the ip responsible for processing slot slot nodes

Port: the port responsible for processing slot slot nodes

For example, MOVED 10086 127.0.0.1 MOVED 7002 indicates that the key-value pair requested by the client is actually on the instance of 127.0.0.1 virtual 7002 in the hash slot 10086.

The returned MOVED command is equivalent to telling the client about the new instance where the hash slot is located.

In this way, the client can connect directly to 7002 and send an operation request.

At the same time, the client updates the local cache to update the corresponding relationship between the slot and the Redis instance correctly.

When the redis-cli client in cluster mode receives the MOVED error, it does not print out the MOVED error, but automatically redirects the node according to the MOVED error and prints the steering information, so we cannot see the MOVED error returned by the node. Redis-cli clients that use stand-alone mode can print MOVED errors.

In fact, Redis tells the client to redirect access to the new instance in two cases: MOVED and ASK. Let's analyze the use of the ASK redirect command.

Re-slice

In a cluster, the correspondence between instances and hash slots is not static. There are two most common changes:

In the cluster, instances are added or deleted, and Redis needs to reassign hash slots.

In order to load balance, Redis needs to redistribute the hash slots across all instances.

Resharding can be carried out online, that is, during the resharding process, the cluster does not need to be offline.

For example, as mentioned above, we have formed a cluster of 7001, 7002, and 7003 nodes, and we can add a new node 127.0.0.1 to this cluster.

$redis-cli-c-p 7001127.0.0.1 7004OK > CLUSTER MEET 127.0.0.1 7004OK

Then, by re-slicing, the slot 15001 ~ slot 16383 originally assigned to node 7003 is changed to be assigned to 7004.

During the period of resharding, during the process of migrating slots from the source node to the destination node, there may be such a situation: what if some of the data in a slot is migrated to a new instance and some are not migrated?

When this migration is partially complete, the client receives an ASK error message.

ASK error

If the client sends a command related to the database key to the target node, and the key to be processed by this command happens to belong to the slot being migrated:

The source node first looks for the specified key in its own database and, if found, executes the command directly.

Conversely, if the source node is not found, then the key may have been migrated to the target node, and the source node will send an ASK error to the client, directing the client to the target node and re-sending the command to be executed before.

It seems a little complicated, let's give an example to explain.

As shown in the figure above, node 7003 is migrating slot 16383 to 7004, which contains hello and world, where the key hello remains on node 7003, while world has migrated to 7004.

We send a command about hello to node 7003, which is executed directly:

127.0.0.1 you get the key 7003 > GET "hello"you get the key 'hello'"

If we send world to node 7003, then the client will be redirected to 7004:

127.0.0.1 ASK 7003 > GET "world"-> (error) error 7004

After receiving an ASK error, the client first sends an ASKING command, and then sends the GET "world" command.

The ASKING command is used to open the ASKING identity of a node, and the command cannot be executed until it is opened.

The difference between ASK and MOVED

Both ASK and MOVED errors can cause client redirection, the difference being:

The MOVED error represents that the responsibility for the slot has been transferred from one node to another: after the client receives a MOVED error about slot I, every time the client encounters a command request about slot I, the client can send the command request directly to the node pointed to by the MOVED error, because that node is currently responsible for slot I.

ASK is only a temporary measure in the process of two nodes migrating slots: after the client receives the ASK error about slot I, the client will only send the command request about slot I to the node pointed to by the ASK error in the next command request, but if the client requests the data in slot I again, it will still send the request to the node that was originally responsible for slot I.

That is, the ASK command only allows the client to send a request to the new instance, and does not update the hash slot allocation information cached by the client. Unlike the MOVED command, the local cache is changed so that all subsequent commands are sent to the new instance.

We now know how to implement a Redis cluster. Let's analyze again, how does the Redis cluster achieve high availability?

Replication and failover

The nodes in the Redis cluster are also divided into master nodes and slave nodes.

The master node is used for the processing slot

The slave node is used to copy the master node, and if the replicated master node goes offline, it can continue to provide services in place of the master node.

For example, for a cluster with four primary nodes from 7001 to 7004, you can add two nodes: 7005, 7006. And set these two nodes as 7001 slave nodes.

Set the slave node command:

CLUSTER REPLICATE

As shown in the figure:

If the master node 7001 goes offline at this time, the remaining working master node in the cluster will choose one of the two slave nodes in 7001 as the new master node.

For example, if node 7005 is selected, the slot originally handled by node 7001 will be handed over to node 7005 for processing. Node 7006 will copy the new primary node 7005 instead. If the subsequent 7001 comes back online, it will become the slave node of 7005. As shown in the following figure:

Fault detection

Each node in the cluster periodically sends PING messages to other nodes to detect whether the other node is online. If the party receiving the message does not return the PONG message within the specified time, the party receiving the message will be marked as "suspected offline" by the sender.

The nodes in the cluster exchange the status information of each node by sending messages to each other.

Three states of the node:

Online statu

Suspected offline status PFAIL

Offline status FAIL

Just because a node thinks that a node is missing does not mean that all nodes think it is missing. In a cluster, more than half of the master nodes responsible for processing slots believe that a master node is offline, and the cluster believes that the node needs to be switched between master and slave.

Redis cluster nodes use Gossip protocol to broadcast their own state and their cognitive changes to the whole cluster. For example, if a node finds that a node has lost contact (PFail), it will broadcast this message to the entire cluster, and other nodes can receive this missing message.

As we all know, the Sentinel mechanism can achieve automatic failover by monitoring, automatically switching the main library, and notifying the client. So how does Redis Cluster achieve automatic failover?

Fail-over

When a slave node finds that the master node it is replicating has entered the "offline" state, the slave node will begin to fail over the offline master node.

Steps to perform the failover:

In all the slave nodes of the copy offline master node, select a slave node

The selected slave node executes the SLAVEOF no one command to become the master node

The new master node will revoke all slot assignments to the offline master node and assign all these slots to itself.

The new master node broadcasts a PONG message to the cluster to let other nodes in the cluster know that the node has changed from a slave node to a master node and has taken over the slot responsible for the original master node.

The new master node begins to receive command requests related to its own processing slot, and the failover is completed.

Choose the master

This selection method is very similar to that of the Sentinel, both based on the lead algorithm of the Raft algorithm. The process is as follows:

The configuration era of the cluster is a self-increasing counter with an initial value of 0

When a node in the cluster starts a failover operation, the cluster configuration era is added by 1

For each configuration era, each master node responsible for the processing slot in the cluster has a chance to vote, and the first slave node that asks for a vote from the master node will get the vote from the master node.

When the slave node finds that its copied master node has entered the "offline" state, it will broadcast a message to the cluster requesting to receive the message, and the master node with the right to vote will vote for itself.

If a master node has the right to vote and has not yet voted for another slave node, the master node will return a message to the slave node requesting the vote, indicating that it supports the slave node to become the new master node.

Each slave node participating in the election will calculate how many master nodes have been supported.

If there are N master nodes with voting rights in the cluster, when the support vote received by a slave node is greater than or equal to Nmax 2 + 1, the slave node will be elected as the new master node

If a sufficient number of votes are not collected from nodes in a configuration era, the cluster enters a new configuration era and chooses the host again.

Message

Each node in the cluster communicates by sending and receiving messages. The node that sends the message is called the sender, and the node that receives the message is called the receiver.

There are five main types of messages sent by nodes:

MEET message

PING message

PONG message

FAIL message

PUBLISH message

Each node in the cluster exchanges the status information of different nodes through the Gossip protocol. Gossip is composed of MEET, PING and PONG messages.

Every time the sender sends MEET, PING and PONG messages, he or she will randomly select two nodes (either master or slave) from the list of known nodes and send them to the receiver.

When the receiver receives the MEET, PING and PONG messages, he or she processes them differently according to whether he or she knows the two nodes:

If the selected node does not have a list of known nodes, it is the first contact, and the receiver will communicate according to the ip and port number of the selected node.

If it already exists, the communication has been completed before, and then the information of the original selected node will be updated.

This is the end of the introduction to "what is a Redis High availability Cluster". Thank you for your reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.