Shulou(Shulou.com)06/01 Report--

The problem of redis Sentinel + Master and Slave

Suppose we configure 200 GB of memory on a master-slave machine, but when the business requirement is 500 GB, the master-slave structure + sentinel can achieve high availability failover + redundant backup, but it can not solve the problem of data capacity. With Sentinel, every instance of redis is fully stored, and the content of each redis is complete data, which is a waste of memory and has bucket effect.

In order to maximize the use of memory, you can use cluster clustering, that is, distributed storage. That is, each redis stores different content.

There are generally two kinds of Redis distributed schemes. Client partition scheme

The advantage is that the partition logic is controllable, while the disadvantage is that you need to deal with data routing, high availability, failover and other problems. For example, before redis2.8, the common practice is to obtain the hashcode of a key, and then distribute the rest to different nodes, but this approach does not support dynamic scalability requirements well. Once nodes are added or deleted, key will not be hit in the redis.

two。 Agency scheme

The advantages are simplified client-side distributed logic and convenient upgrade maintenance, while the disadvantages are increasing the complexity of architecture deployment and performance loss, such as twemproxy and Codis.

3. The official provides us with a proprietary cluster scheme: Redis Cluster

It is very elegant to solve the problem of Redis cluster, easy to deploy, so understanding the application of Redis Cluster will greatly liberate the workload of using distributed Redis.

Introduction to Redis Cluster

Redis Cluster is a distributed solution of Redis, which is officially launched in version 3.0, which effectively solves the needs of Redis distribution. When you encounter bottlenecks such as stand-alone memory, concurrency and traffic, you can use Cluster architecture to achieve load balancing.

Architecture diagram

In this figure, each blue circle represents a redis server node. Any two of them are connected to each other. The client can connect to any node, and then access any node in the cluster for access and other operations.

Redis clustering provides the following two benefits:

The ability to automatically split data into multiple nodes

When some nodes in the cluster fail or are unable to communicate, they can still continue to process command requests and have the ability to fail over automatically.

Redis cluster vs. Replication + sentinal how to choose

If you have a small amount of data, mainly carrying high-concurrency and high-performance scenarios, for example, your cache is usually only a few gigabytes, and a single machine is enough.

Replication: one mater, multiple slave. Several slave are related to your required read throughput. Combine with sentinal cluster to ensure the high availability of redis master-slave architecture.

Redis cluster: mainly for scenarios of massive data + high concurrency + high availability, massive data. If you have a large amount of data, it is recommended to use redis cluste.

How does data distribution work? what is data distribution? There are two ways to distribute data, sequential partition and hash partition

The distributed database should first solve the problem of mapping the whole data set to multiple nodes according to the partition rules, that is, dividing the data set into multiple nodes, and each node is responsible for a subset of the overall data.

Sequential distribution

Sequential distribution is to spread a whole piece of data among many machines, as shown in the following figure.

The sequential distribution is generally evenly distributed.

Hash partition

As shown in the following figure, the whole number 1x 100 is the number generated by the remainder through the function of hash. This ensures that the string of numbers is fully scattered and evenly distributed to each machine.

Hash distribution and sequential distribution are only applicable in the scenario. Hash distribution cannot be accessed sequentially. For example, if you want to access 1x100, hash distribution can only traverse all data. At the same time, hash distribution has nothing to do with business data because it has done hash.

Data skew, data Migration and Node scaling

Sequential distribution can lead to skewed data, mainly the tilt of access. Each click will focus on visiting a certain machine, which leads to the final data on this machine, which is the biggest disadvantage of sequential distribution.

But there is actually a problem with hash distribution. When we want to expand the machine, it is professionally called "node scaling". At this time, because it is a hash algorithm, it will lead to data migration.

Hash partition mode

Because redis-cluster uses hash partition rules, analyze the following partition forms

1. Node remainder partition

Using specific data (including redis keys or user ID), and then according to the number of nodes N, use the formula: hash (key)% N to calculate a value of 0 ~ (Nmuri 1), which is used to determine which node the data is mapped to. That is, the hash value takes the remainder of the total number of nodes. The remainder x indicates that this piece of data is stored on the 1st node.

Disadvantages: when the number of nodes changes (expanding or shrinking), the mapping between the data and the nodes needs to be recalculated, so that according to the new rules, either the previously stored data cannot be found, or the previously stored data is remapped to the new node (resulting in data migration of previously stored data)

Practice: the rules of database partition and table partition, which are commonly used in database, generally use the way of pre-partition to plan the number of partitions in advance according to the amount of data, such as dividing them into 512 or 1024 tables to ensure that the amount of data can be supported for a period of time in the future. then migrate the table to other databases according to the load

2. Consistent hash

The idea of consistent hash partition (Distributed Hash Table) is to assign a token to each node in the system, and the range is generally 0,232. These token form a hash ring. When data read and write performs a node lookup operation, the hash value is first calculated according to key, and then the first token node greater than or equal to the hash value is found clockwise

The figure above is the principle analysis of a consistent hash.

Suppose we have four n1~n4 machines, and we assign a unique token to each machine, and each time there is data (× × represents data in the figure). The consistent hash algorithm stipulates that the data is drifted clockwise every time, that is, the data of × × in the figure points to N3.

At this time, we need to add a node N5, between N2 and N3, the data will still drift (will be offset to nodes greater than or equal to), but at this time, have you noticed that only this part of the data of n2~n3 is drifted, and other data will not change? the biggest advantage of this way is that adding and deleting nodes only affect the neighboring nodes in the hash ring and have no effect on other nodes.

Disadvantages: the load of each node is different, because the hash of each node is calculated according to key, in other words, it is assumed that there are enough key, which is scattered uniformly by the hash algorithm, but there are too few nodes, so that the number of key processed by each node is different, or even very different, which will lead to a lot of pressure on some nodes.

Practice: adding and subtracting nodes will cause part of the data in the hash ring to miss, which needs to be manually processed or ignored, so consistent hashes are often used to cache scenarios.

3. Virtual slot partition

Virtual slot partition skillfully uses hash space, using well-dispersed hash functions to map all data to a fixed range of integers, which are defined as slots (slot). This range is generally much larger than the number of nodes, for example, the Redis Cluster slot range is 0mm 16383. Slot is the basic unit of data management and migration in the cluster. The main purpose of adopting large-scale slots is to facilitate data splitting and cluster expansion. Each node is responsible for a certain number of slots, as shown in the following figure.

The current cluster has 5 nodes, and each node is responsible for an average of about 3276 slots. Due to the use of high-quality hashing algorithm, the data mapped by each slot is usually uniform, and the data is equally divided into 5 nodes for data partition. Redis Cluster uses virtual slot partitioning. The Redis data partitioning method is introduced below.

Whenever key visits, the Redis Cluster accountant calculates whether the hash is in this range. They both know which machine the corresponding slots are on, so that they can distribute them evenly.

Set up the cluster manually

After introducing the Redis cluster partition rules, let's start building a Redis cluster. Building a cluster requires the following three steps:

Prepare node handshake allocation slot

Node planning container name container IP address mapping port number service operation mode redis-master10172.50.0.106400- > 6400, 16400-> 16400 master11172.50.116401-> 6401, 16401-> 16401 master122.50.126402-> 6402 masterredisslave10172.30.0.106403-> 6403 slave11172.30.0.116404-> 6404slave12172.30.126405-> 6405slave1. Prepare the node

Redis cluster is generally composed of multiple nodes, and the number of nodes is at least 6 to ensure the formation of a complete and highly available cluster. Each node needs to enable the configuration cluster-enabled yes to make Redis run in cluster mode, and the above configuration is given to the configuration file of redis and started accordingly.

Other configurations are consistent with stand-alone mode. The configuration file naming rule redis- {port} .conf is ready to start all nodes. If there is no cluster configuration file when starting for the first time, it will automatically create one. The file name is controlled by the cluster-config-file parameter item. It is recommended to use node- {port} .conf format, that is to say, there will be two configuration files.

When the information of the nodes in the cluster changes, such as adding nodes, offline nodes, failover and so on. The node automatically saves the cluster state to the configuration file. It should be noted that Redis automatically maintains cluster configuration files and does not modify them manually to prevent cluster information confusion when nodes are restarted.

The profile information is as follows:

The content of the file records the initial state of the cluster. The most important thing here is the node ID, which is a 40-bit hexadecimal string that uniquely identifies a node in the cluster. The node ID is created only once during the initialization of the cluster. When the node is restarted, the cluster configuration file will be loaded for reuse, combined with the corresponding cluster operations, and the running ID of Redis will change each time it is restarted.

Dockerfile information used to build an image

FROM centos:latestMAINTAINER peter "ttone2@vip.qq.com" RUN groupadd-r redis & & useradd-r-g redis redisRUN yum-y update & & yum-y install epel-release\ & & yum-y install redis & & yum-y install wget\ & & yum-y install net-tools\ & & yum-y install ruby & yum-y install rubygemsRUN wget https://rubygems.org/downloads/redis-3.2.1.gem & & gem install-l. / redis- 3.2.1.gem\ & & rm-f redis-3.2.1.gemCOPY. / config/redis-trib.rb / usr/binCOPY. / config/redis.sh / usr/binRUN mkdir-p / config & & chmod 775 / usr/bin/redis.sh & & chmod 775 / usr/bin/redis-trib.rb

Create a separate network

Master node network card docker network create-d bridge-- subnet 172.50.0.0 docker network create 16 redis-cluster_redis-master slave node network card docker network create-d bridge-- subnet 172.30.0.0 Universe 16 redis-cluster_redis-salve

Execute the New Container command in turn:

Docker run-itd-- name redis-master10-v / home/work/redis-cluster/config/:/config-- workdir / config-- net redis-cluster_redis-master-e PORT=6400-p 6400 redis-clusterdocker run 16400-- ip 172.50.0.10 redis-clusterdocker run-itd-- name redis-master11-v / home/work/redis-cluster/config/:/config-- workdir / config-- net redis-cluster_redis-master-e PORT=6401-p 6401 Vane 6401-p 1640116401 -- ip 172.50.0.11 redis-cluster...docker run-itd-- name redis-slave11-v / home/work/redis-cluster/config/:/config-- workdir / config-- net redis-cluster_redis-slave-e PORT=6404-p 6404 redis-clusterdocker run 16404-- ip 172.30.0.11 redis-clusterdocker run-itd-- name redis-slave12-v / home/work/redis-cluster/config/:/config-- workdir / config-- net redis-cluster_ Redis-slave-e PORT=6405-p 6405 ip 16405-- ip 172.30.0.12 redis-cluster

Entrypoint and CMD commands cannot be executed because they are created using a command line, so you need to go to each machine to start the Redis service.

[root@izuf64gdegum84eku07pljz config] # docker exec-it redis-master11 bash [root@c101670ac2f4 config] #. / redis.sh & 2. Node handshake

Node handshake refers to the process that a group of nodes running in cluster mode communicate with each other through Gossip protocol to perceive each other. The node handshake is the first step for the cluster to communicate with each other. The client initiates the command: cluster meet {ip} {port}

About Gossip, you can see the introduction of the article.

Let nodes 6379 and 6380 communicate by shaking hands with the command cluster meet 127.0.0.1 6380. The cluster meet command is an asynchronous command that returns immediately after execution. Internally initiate handshake communication with the target node

1) Node 6379 locally creates 6380 node information objects and sends meet messages

2) after receiving the meet message, node 6380 saves the 6379 node information and replies to the pong message

3) after that, nodes 6379 and 6380 regularly communicate with each other through ping/pong messages

[root@63d7fd48dc81 config] # redis-cli-p 6400127.0.0.1 6401OK127.0.0.1:6400 > cluster meet 47.101.139.0 6401OK127.0.0.1:6400 > cluster meet 47.101.139.0 6402OK127.0.0.1:6400 > cluster meet 47.101.139.0 6403OK127.0.0.1:6400 > cluster meet 47.101.139.0 6404OK127.0.0.1:6400 > cluster meet 47.101.139.0 6405OK127.0.0.1:6400 >

Confirm that the six nodes are aware of each other and form a cluster through the cluster nodes command

Note:

1. Each Redis Cluster node occupies two TCP ports. One listens to client requests, which is 6379 by default, and the other adds 10000 to the previous port, such as 16379, to listen for data requests. Nodes and nodes will listen on the second port and use a set of binary protocols to communicate.

Nodes will use a set of protocols for failure detection, configuration updates, failover authentication and so on.

In order to ensure normal access between nodes, we need to pay attention to the configuration of the firewall.

2. After the node establishes the handshake, the cluster does not work properly. At this time, the cluster is offline and all data reading and writing are prohibited.

3. Set up slave node

As a complete cluster, master and slave nodes are needed to ensure that it can fail over automatically when it fails. In cluster mode, the role of Reids node is divided into master node and slave node.

The node started for the first time and the node of the assigned slot are the master node, and the slave node is responsible for copying the master node slot information and related data.

Use the cluster replicate {nodeId} command to make a node a slave. Where the command execution must be executed on the corresponding slave node, and the current node is set as the slave node of the node specified by node_id

[root@izuf64gdegum84eku07pljz config] # redis-cli-h 47.101.139.0-p 6403 cluster replicate 1a30f25427cf9ca0a8362a50e6a313b933fe7df2 [root@izuf64gdegum84eku07pljz config] # redis-cli-h 47.101.139.0-p 6404 cluster replicate b99ed66e154468f0d2fc21ff4d7113bf0424262b [root@izuf64gdegum84eku07pljz config] # redis-cli-h 47.101.139.0-p 6405 cluster replicate ba4547686e34ab121a8226291eb5ab019c53c5f8

Results:

127.0.0.1 cluster nodes056b42451d574aa383464a4cb1b50b1511d6ebe5 47.101.139.0 cluster nodes056b42451d574aa383464a4cb1b50b1511d6ebe5 6404 slave b99ed66e154468f0d2fc21ff4d7113bf0424262b 01555686394258 4 connectedba4547686e34ab121a8226291eb5ab019c53c5f8 47.101.139.0 master-01555686390247 0 connected2a2676234dcd3b6991dd4491e61feebbdb689c69 47.101.139.0 cluster nodes056b42451d574aa383464a4cb1b50b1511d6ebe5 6403 slave 1a30f25427cf9ca0a8362a50e6a313b933fe7df2 0155568394759 3 connected1a30f25427cf9ca0a8362a50e6a313b933fe7df2 47.101.139.0 master 6400 master-0155568395260 1 connected0dc1306db8bde9868846dacc631c80f6abe85ff3 47.101.139.0 connectedb99ed66e154468f0d2fc21ff4d7113bf0424262b 172.50.11 myself,master-00 2 connected4. Distribution slot

The Redis cluster maps all the data to 16384 slots. Each key is mapped to a fixed slot, and only when the node assigns slots can it respond to the key commands associated with those slots. Assign slots to nodes through the cluster addslots command

Use the bash feature to set the slot (slots) in batches, and the command is as follows:

[root@izuf64gdegum84eku07pljz config] # redis-cli-h 47.101.139.0-p 6400 cluster addslots {0.5461} [root@izuf64gdegum84eku07pljz config] # redis-cli-h 47.101.139.0-p 6401 cluster addslots {5462... 10922} [root@izuf64gdegum84eku07pljz config] # redis-cli-h 47.101.139.0-p 6402 cluster addslots {10923. 16383}

Slot information of the primary node:

127.0.0.1master 6401 > cluster nodes056b42451d574aa383464a4cb1b50b1511d6ebe5 47.101.139.0 connectedba4547686e34ab121a8226291eb5ab019c53c5f8 47.1055687118050 4 connectedba4547686e34ab121a8226291eb5ab019c53c5f8 47.101.139.0 master-01555687118550 0 connected 10923-163832a2676234dcd3b6991dd4491e61feebbdb689c69 47.101.139.0 connected 6403 slave 1a30f25427cf9ca0a8362a50e6a313b933fe7df2 01555687115546 3 connected1a30f25427cf9ca0a8362a50e6a313b933fe7df2 47.101.139.0 slave ba4547686e34ab121a8226291eb5ab019c53c5f8 01555687117549 1 connected 0-54610dc1306db8bde9868846dacc631c80f6abe85ff3 47.101.139.0 slave ba4547686e34ab121a8226291eb5ab019c53c5f8 01555687116547 5 connectedb99ed66e154468f0d2fc21ff4d7113bf0424262b 172.50.0.11 Switzerland 6401 myself,master-00 connected 5462-10922

We manually set up a cluster according to the Redis protocol. It consists of six nodes, three master nodes are responsible for processing slots and related data, and three slave nodes are responsible for failover. Manually building a cluster is easy to understand the process and details of cluster establishment, but we find that cluster construction requires many steps. When there are many cluster nodes, it is bound to increase the complexity and operation and maintenance costs of building a cluster. Therefore, Redis officially provides redis-trib.rb tools to facilitate us to quickly build clusters.

5. Operation cluster

-c cluster mode

[root@izuf64gdegum84eku07pljz config] # redis-cli-c-h 47.101.139.0-p 6400 set mayun alibaba-996 [root@izuf64gdegum84eku07pljz config] # redis-cli-c-h 47.101.139.0-p 6400 get mayun "alibaba-996"

If no cluster mode is specified, the following error occurs

[root@izuf64gdegum84eku07pljz config] # redis-cli-h 47.101.139.0-p 6400 set mayun alibaba-996 (error) MOVED 11165 47.101.139.0 set mayun alibaba-996 6402

All commands:

CLUSTER info: print information about the cluster CLUSTER nodes: list information about all nodes (node) currently known in the cluster CLUSTER meet: add the nodes specified by ip and port to the cluster CLUSTER addslots [slot...]: assign one or more slot to the current node CLUSTER delslots [slot...]: remove one or more slots to the current node CLUSTER slots: list slots, Node information CLUSTER slaves: list the slave node information under the specified node CLUSTER replicate: set the current node as the slave node of the specified node CLUSTER saveconfig: manually execute the command to save the configuration file of the cluster By default, the cluster automatically saves the configuration file CLUSTER keyslot when the configuration is modified: list the slot in which the key is placed CLUSTER flushslots: remove all slots assigned to the current node, making the current node a node without any slot assigned CLUSTER countkeysinslot: return the number of key value pairs currently contained in the slot CLUSTER getkeysinslot: return the key CLUSTER setslot node in the count slot to assign the slot to the specified node, if the slot has been assigned to another node Then let another node delete the slot, and then assign CLUSTER setslot migrating to migrate the slot of this node to the specified node CLUSTER setslot importing import slot slot from the node specified by node_id to this node CLUSTER setslot stable cancel the import of slot slot (import) or migrate (migrate) CLUSTER failover: manually failover CLUSTER forget: remove the specified node from the cluster, so the handshake cannot be completed, and the expiration is 60s. 60 seconds later, the two nodes will continue to complete the handshake CLUSTER reset [HARD | SOFT]: reset the cluster information. Soft is to clear the information of other nodes, but if you do not modify your own id,hard, you will also modify your own id. If you do not pass this parameter, you will use soft to CLUSTER count-failure-reports: list the length of the fault report of a node CLUSTER SET-CONFIG-EPOCH: set the node epoch. Only before the node joins the cluster can you set redis-trib.rb to build the cluster.

Redis-trib.rb is a Redis cluster management tool implemented by Ruby. Internally, Cluster-related commands help us simplify common OPS operations such as cluster creation, inspection, slot migration and balancing. You need to install the Ruby dependent environment before using it. Here are the detailed steps for setting up a cluster

Internally, Cluster-related commands help us simplify common operation and maintenance operations such as cluster creation, inspection, slot migration and balancing. Before use, you need to install the Ruby dependent environment. The related extension has been written in the dockerfile file. Just check and understand the meaning.

After 6 nodes are started, use the redis-trib.rb create command to complete the node handshake and slot allocation process

Redis-trib.rb create-- replicas 1 47.101.139.0 Viru 6400 47.101.139.0 Viru 6401 47.101.139.0 Viru 6402 47.101.139.0 6403 47.101.139.0 Vol 6404 47.101.139.0 Vol 6405

-- the replicas parameter specifies that each master node in the cluster is equipped with several slave nodes. If you set this parameter to 1, the master and slave nodes will not be assigned under the same machine as far as possible, so the list of nodes will be reordered. The order of the node list is used to determine the master and slave roles, first the master node and then the slave node. During the creation process, a plan for the role assignment of the master and slave nodes is first given, and a report is generated.

Command description: redis-trib.rb helpUsage: redis-trib # create cluster create host1:port1. HostN:portN-replicas # with this parameter indicates whether there are slaves. Arg indicates the number of slaves. # check cluster check host:port# to view cluster information info host:port# repair cluster fix host:port-- timeout # online migration slot reshard host:port# is a required parameter It is used to obtain the entire cluster information from one node, which is equivalent to the entrance to get the cluster information-- from # which source nodes need to migrate slot, which can be migrated from multiple source nodes, separated by commas, passing the node's node id, and directly passing-- from all, so that the source node is all the nodes of the cluster, if this parameter is not passed. The user will be prompted to enter the node id of the destination node that to # slot needs to migrate during the migration process. Only one can be entered in the destination node. If this parameter is not passed, the user will be prompted for input during the migration process. -- the number of slot to be migrated in slots #. If this parameter is not passed, the user will be prompted for input during the migration process. -- yes # sets this parameter. When printing and executing the reshard plan, you can prompt the user to enter yes confirmation and then execute reshard-- timeout # to set the timeout of the migrate command. -- pipeline # defines the number of key fetched by the cluster getkeysinslot command at a time. If not passed, the default value is 10. # balance the number of slot of cluster nodes rebalance host:port-- weight-- auto-weights-- use-empty-masters-- timeout-- simulate-- pipeline-- threshold # add new nodes to the cluster add-node New_host:new_port existing_host:existing_port-- slave-- master-id # removes nodes from the cluster del-node host:port node_id# sets the timeout for heartbeat connections between cluster nodes set-timeout host:port milliseconds# executes the command call host:port command arg arg on all nodes in the cluster. Arg# imports external redis data into cluster import host:port-- from-- copy-- replace

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.