Shulou(Shulou.com)06/02 Report--

This article mainly explains "what is the principle of Redis cluster". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the principle of Redis cluster"?

Redis, as an in-memory database with excellent performance, supports Weibo 100 million social platform, and has become a standard for many Internet companies. Here, we will take the Redis Cluster cluster as the core, based on the latest Redis5 version, from the principle to the actual combat, play with the Redis cluster

Common Redis cluster schemes

Before introducing Redis Cluster clustering solutions, to facilitate comparison, let's take a brief look at the common Redis clustering schemes in the industry:

1 based on client-side fragmentation

Redis Sharding is a multi-Redis instance clustering method widely used in the industry before Redis Cluster came out. The main idea is to slice the Redis data according to the key hash value of the hash data and map the data to their respective nodes based on the hash algorithm.

The advantage is that it is easy to implement, and the disadvantage is that when the Redis cluster adjusts, each client needs to update and adjust.

2 slicing based on proxy server

The client sends a request to the independently deployed agent component, which parses the client's data, forwards the request to the correct node, and finally returns the result to the client

The advantage lies in transparent access and easy cluster expansion, while the disadvantage lies in the loss of performance due to the addition of a layer of proxy forwarding.

3 Redis Sentinel (sentinel)

Redis Sentinel is a highly available solution officially provided from Redis version 2.6. on the basis of Redis master-slave replication cluster, Sentinel cluster is added to monitor the entire Redis cluster. When the Redis cluster master node fails, Sentinel fails over and elects a new master. At the same time, Sentinel itself supports highly available cluster deployment.

The advantage is that the cluster is highly available and high-performance read and write, but the disadvantage is that there is no data fragmentation, each node needs to carry the complete data set, and the load capacity is limited by being a Redis server. It only supports vertical expansion by increasing machine memory, but not horizontal expansion.

Redis Cluster Design 1 overall Design

Redis Cluster is a high availability cluster scheme officially launched in version 3.0. compared with Redis Sentinel,Redis Cluster scheme, it does not need to deploy Sentinel cluster, but realizes cluster monitoring through cluster internal communication and master-slave switching in case of failure; at the same time, it supports internal data fragmentation based on hash and dynamic horizontal expansion.

The overall structure is as follows:

There are multiple master nodes in the cluster, and each master node has multiple slave nodes. The data between master and slave nodes are consistent. At least three master nodes are needed, and each master node needs at least one slave node.

High availability: automatic master-slave switching when master node fails

High performance: master node provides read-write service, slave node read-only service, improving system throughput

Scalability: the data of the cluster is stored in fragments, and the data between the main nodes is different, and the corresponding data is maintained respectively. You can add nodes for the cluster to expand capacity, or you can scale down some nodes horizontally.

2 data slicing

The whole data set is allocated to multiple nodes according to certain rules, which is called data sharding. The slicing scheme adopted by Redis Cluster is hash sharding.

The basic principle is as follows: Redis Cluster first defines the interval numbered 0 ~ 16383, which is called slot, and all keys are mapped to 0 ~ 16383 integer slots according to the hash function. The calculation formula: slot=CRC16 (key) & 16383. Each node is responsible for maintaining part of the slot and the key value data mapped by the slot.

Slot is the basic unit of Redis cluster management data. Cluster expansion and contraction is the movement of slot and data between nodes.

The mapping relationship between slots and nodes is as follows:

Each cluster node maintains a 16384 bit (2kB) bit array, with each bit corresponding to the same numbered slot, using 0 / 1 to identify whether it has its own for a slot.

The cluster node also maintains the mapping from slot to cluster node, which is an array of 16384 in length, the array subscript represents the slot number and the value is the node information.

(3) Cluster expansion

Redis Cluster supports dynamic expansion or reduction of cluster capacity without affecting the external service of the cluster. When a new Redis node joins the existing cluster, it needs to migrate slots and data for it, ensuring that each node is responsible for a similar number of slots after migration, so that the data is evenly distributed on each node.

The whole data migration involves a series of operations. Redis provides cluster management tools, including Ruby-based redis-trib.rb and Redis5's newly provided C-based redis-cli. The following introduction takes redis-cli as an example.

The source node migrates the specified slot data to the destination node. The basic process is as follows:

(1) redis-cli sets the target node to specify the slot status importing, so that the target node is ready to migrate to slot data

(2) redis-cli sets the source node to specify the slot status migrating, so that the source node is prepared to move out of the slot data

(3) redis-cli batch migrates the data from the source node specified in slot to the target node

(4) after data migration, redis-cli notifies all master nodes of the cluster that slots are assigned to the target node, and the master node updates the mapping information between slot and nodes.

In general, if the data requested by the client is not on the node, the node will reply to the MOVED redirection message, and the client will request the correct node based on this information. For the slot data being migrated, the design to ensure that the client can still access it normally is as follows:

(1) the mapping information between slot and nodes is not updated until the migration is completed. If the mapping information in progress is the same as that before the migration.

(2) if the client accesses the source node and the accessed key has not moved out, the key will be processed normally.

(3) if the client accesses the source node, the accessed key has moved out, and the source node returns ASK redirection information

(4) the client extracts the information of the target node according to the ASK redirection exception, first sends the ASKING command to the target node to request operation, and then executes the key command.

There are the following differences between ASK and MOVED redirection controls:

ASK redirection indicates that the cluster is undergoing slot data migration, and the client cannot know when the migration will be completed, so it can only be a temporary redirection. The client will not update the mapping cache from slot to Redis nodes.

The slot corresponding to the MOVED redirection key has been explicitly assigned to the new node, so the mapping cache from slot to Redis node needs to be updated.

4 CAP trade-off

CAP includes: consistency (Consistency), availability (Availability), partition fault tolerance (Partition tolerance). If the system cannot achieve data consistency within the time limit, it means that the partition situation has occurred and a choice must be made between C and A.

Redis Cluster chooses AP architecture. In order to ensure availability, Redis does not guarantee strong consistency. Under certain conditions, there will be data inconsistency or even loss of write operations.

The first reason is that in order to make a tradeoff between performance and consistency, data synchronization between master and slave nodes is replicated asynchronously. When the client successfully writes to the master node and master returns successfully, the master node replicates the write operation asynchronously to the slave node.

Another reason is that when a cluster sends a network partition, the cluster may be divided into two parts: the majority and the minority, if the masterA node is in the minority, and if the network partition occurs for a short time, the cluster will continue to operate normally; if the partition is long enough for the majority to elect the new master instead of matsterA, then the data written to masterA during the partition will be lost

During the network partition, the maximum time that a client can send write commands to matsterA is limited, which is called node timeout (cluster-node-timeout), and is an important configuration option for Redis clusters.

Thank you for your reading, the above is the content of "what is the principle of Redis cluster". After the study of this article, I believe you have a deeper understanding of what the principle of Redis cluster is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.