Shulou(Shulou.com)06/01 Report--

This article mainly shows you the "sample analysis of Redis sharding mechanism", which is easy to understand and clear. I hope it can help you solve your doubts. Let the editor lead you to study and study the "sample analysis of Redis sharding mechanism".

Introducing the concept of slicing-- the function of slicing mechanism

This article mainly talks about the slicing mechanism of Redis. If there is no slicing mechanism, Redis will be limited to the memory capacity supported by a single computer. Redis's slicing mechanism allows data to be split and stored on different Redis instances, and each Redis instance contains only a subset of all keys. It can reduce the pressure of a single Redis and improve the scalability and computing power of Redis. If we only use one Redis instance, the service will be stopped directly when Redis goes down, so we can adopt the sharding mechanism to change the data maintained by one Redis instance to that maintained by multiple Redis instances.

Slicing scheme

(1) range slicing

Sharding needs to map different key to different Redis instances for storage, so the mapping rules of key need to develop an algorithm, and the simplest sharding scheme should be range sharding. Range fragmentation is easy to understand. For example, we store basic user information, and we develop an algorithm to map user user_id from 0 to 1000 to instance A from 1000 to 2000 to instance B. and so on. This scenario is easy to use, but it raises a problem: we need to maintain the relationship between the user_id scope and the mapping instance. It is this problem that leads to the simplicity of range sharding, but its efficiency is much lower than that of other sharding schemes, so range sharding is generally not used as a sharding scheme in Redis.

(2) Hash fragmentation

For example, we currently have four Redis instances, and we need to store one key. We can convert the key name to a long integer number through the hash function crc32 (), and then take the remainder of the long integer number pair 4 to get an example of the mapping. But this allocation scheme also has a drawback: when we need to add or remove Redis instances, it will cause a large number of key to miss. So at this time, there is an advanced form of hash fragmentation-consistent hash. There are three characteristics of consistent hashing:

Key hash results are assigned to different Redis instances as much as possible.

When instances are added or removed, you need to protect the mapped content from being reassigned to the new instance.

The hash of key should be avoided as much as possible.

However, the concept of consistent hash is not used in Redis, but hash slots are introduced. There are 16384 hash slots in the Redis cluster, and then each key converts the key name into a long integer number through the hash function crc16 (), and then takes the remainder of 16384, and finally determines the hash slot stored by the key. Each Redis instance is responsible for maintaining part of the hash slot, and all instances work together to maintain all the hash slots. The most obvious feature of using hash slots is that it is easy to add or remove Redis instances without suspending all Redis instance services.

Sharding implementation

As mentioned in the previous article on master-slave switching, Sentinel mode can achieve high availability and read-write separation, but the disadvantage is that all Redis instances store the same data, so Redis supports cluster mode, which can simply understand cluster as a plug-in for Redis cluster management, through which Redis distributed storage can be realized.

There are generally three ways of data sharding: client sharding, proxy sharding and server sharding.

Client slicing

Definition: the client calculates which Redis instance key needs to map to.

Advantages: the most obvious advantage of client slicing is that it reduces the complexity of the cluster, while there is no correlation between servers, and the client is responsible for implementing data slicing.

Disadvantages: if the client implements sharding, the client needs to know the information of different Redis instances under the current cluster. Dynamic sharding is required when adding Redis instances, and most Redis needs to be restarted to achieve this function.

Proxy slicing

Definition: the client sends the request to the agent, and the agent calculates the cluster instance information that needs to be mapped, then forwards the client's request to the corresponding cluster instance, and then returns the response to the client.

Advantages: the complexity of the client is reduced, and the client does not have to care about the status information of the backend Redis instance.

Disadvantages: there is an intermediate distribution link, so there is some loss of performance.

Server fragmentation

Definition: the client can communicate with any Redis instance in the cluster. When the client accesses an instance, the server calculates which specific Redis instance key should be mapped to for storage. If the mapped instance is not the current instance, the instance actively guides the client to operate on the key corresponding to the instance. This is actually a process of redirection. Instead of forwarding from the current Redis instance to the corresponding Redis instance, the client receives a notification from the server that the mapped Redis instance is redirected to the mapped instance. At present, it is not fully applicable to the production environment.

Advantages: support high availability, any instance has master and slave, master and slave will automatically take over.

Disadvantages: client languages are required to implement the server cluster protocol, but most languages currently have their client implementation versions.

Preslicing

It is clear from the above that the sharding mechanism is very troublesome to add or remove instances, so we can consider starting 32 node instances from the beginning, and when we can add a new Redis server, we can move half of the nodes to the new Redis server. In this way, we only need to start an empty node on the new server, then move the data, configure the new node as the slave node of the source node, then update the ip information of the moved node, then send the slaveof command to the new server to shut down the master-slave configuration, and finally shut down the instances that the old server does not need and restart the client. In this way, we can complete the movement of data with little downtime.

Shortcomings of slicing mechanism

Sharding is run by multiple Redis instances, so if one of the Redis instances goes down, the whole shard cannot be used, so the sharding mechanism cannot achieve high availability.

If there are different key mappings to different Redis instances, it is not possible to intersect the two key or use transactions at this time.

The use of slicing mechanism because it involves multiple instances, data processing is more complex.

Adding or deleting instances in sharding can be complicated, but it can be improved by using pre-sharding technology.

The above is all the content of this article "sample Analysis of Redis slicing Mechanism". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.