Shulou(Shulou.com)06/03 Report--

Bug scenario description:

In redis-cluster clusters under ruby-2.4.6 and redis-4.1.3.gem environments, the bug that occurs when nodes containing data are removed from the redis-cluster cluster. The following is a detailed description of the scenario where bug appears.

Redis-cluster removes nodes:

Like node addition, removing a node also removes the master node and the slave node.

1. Remove the primary node remove the node using the del-node command of redis-trib

Redis-trib del-node 127.0.0.1 7002 ${node-id}

127.0.0.1 7002 is any node in the redis-cluster cluster, and node-id is the primary node to be deleted. Unlike adding nodes, removing the node node-id is required, test deleting the 7002 primary node:

Example: delete the corresponding instance of redis-cluster cluster node f7a95238e3de39b616b93949ec7c9f86d3867d63: 192.168.1.391986

This node happens to have data.

[root@mysql-redis39 log] # / data/soft/redis-4.0.12/src/redis-trib.rb del-node 192.168.1.391986 ERR 1986 f7a95238e3de39b616b93949ec7c9f86d3867d63 > Removing node f7a95238e3de39b616b93949ec7c9f86d3867d63 from cluster 192.168.1.391986 [ERR] Node 192.168.1.391986 is not empty! Reshard data away and try again. [root@mysql-redis39 log] #

Failed to delete the node, indicating that there is data in the node and cannot be deleted from the redis-cluster. Its data needs to be transferred out, that is, it needs to be refragmented like the new master node.

Need to refragment: (remove 4096 hash slots from 192.168.1.39 hash slots 1986 to be2a864214a624789748c7f753377638c6f88751 192.168.1.54 master node)

[root@mysql-redis39 log] # / data/soft/redis-4.0.12/src/redis-trib.rb reshard 192.168.1.39 slots:0 > Performing Cluster Check (using node 192.168.1.39) M: f7a95238e3de39b616b93949ec7c9f86d3867d63 192.168.1.39) 1986 slots:6827-10922 (4096 slots) master 1 additional replica (s) M: be2a864214a624789748c7f753377638c6f88751 192.168.1.54 slots 1986 slots:0-136 5461-6826 master 2 additional replica (s) M: 8db27e432c9ef45fd37bba20e8ca7b71556541a0 192.168.1.39 master 986 slots:1365-5460 (4096 slots) master 1 additional replica (s) M: 0b3963a0be38ea500013ddcaa5a5524801421dd5 192.168.1.182 master 1 additional replica (s) S: 907fedbbf999084893a9dff000701f6a9a92381a 192.168.1.10 slots: (0 slots) slave replicates 0b3963a0be38ea500013ddcaa5a5524801421dd5S: ae6e9b0139735cb5dbc4ebf3ad6e01b3f5420db5 192.168.1.54 1986 slots: (0 slots) slave replicates be2a864214a624789748c7f753377638c6f88751S: 0efd850873fb346fc3c273a9d3aeaac0a9e4d4a8 192.168.1.182 : (0 slots) slave replicates f7a95238e3de39b616b93949ec7c9f86d3867d63S: 7a778a0fb0acaa000d003008f50af430497218f3 192.168.1.105 slots 1986 slots: (0 slots) slave replicates 8db27e432c9ef45fd37bba20e8ca7b71556541a0S: 47fe8159fd5be2e74e823e077d9afe0fd77570c4 192.168.1.54 slots 986 slots: (0 slots) slave replicates be2a864214a624789748c7f753377638c6f88751 [OK] All nodes agree about slots configuration. > > Check for open slots... > Check slots coverage... [OK] All 16384 slots covered.How many slots do you want to move (from 1 to 16384)? 4096What is the receiving node ID? Be2a864214a624789748c7f753377638c6f88751

Do you need to move to all primary nodes or to a single primary node:

Move 4096 slots to 192.168.1.54 node id 1986 and fill in the f7a95238e3de39b616b93949ec7c9f86d3867d63How many slots do you want to move (from 1 to 16384)? 4096What is the receiving node ID? Be2a864214a624789748c7f753377638c6f88751Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs.Source node # 1:f7a95238e3de39b616b93949ec7c9f86d3867d63 # # 192.168.1.39 node id Source node # 2 1:f7a95238e3de39b616b93949ec7c9f86d3867d63... Moving slot 10920 from f7a95238e3de39b616b93949ec7c9f86d3867d63 Moving slot 10921 from f7a95238e3de39b616b93949ec7c9f86d3867d63 Moving slot 10922 from f7a95238e3de39b616b93949ec7c9f86d3867d63Do you want to proceed with the proposed reshard plan (yes/no)? yes

After confirmation, the card slot of 192.168.1.39 virtual 1986 will be moved to 192.168.1.54 virtual 1986.

[root@mysql-redis39] # / data/soft/redis-4.0.12/src/redis-trib.rb reshard 192.168.1.182 > > Performing Cluster Check (using node 192.168.1.182) M: 0b3963a0be38e0013ddcaa5a5524801421dd5 192.168.1.182 1986 slots:12288-16383 (4096 slots) master 1 additional replica (s) S: 0efd850873fb346fc3c273a9d3aeaac0a9e4d4a8 192.168.1.182 slots: (0 slots) slave replicates f7a95238e3de39b616b93949ec7c9f86d3867d63S: 7a778a0fb0acaa000d003008f50af430497218f3 192.168.1.105: 1986 slots: (0 slots) slave replicates 8db27e432c9ef45fd37bba20e8ca7b71556541a0S: ae6e9b0139735cb5dbc4ebf3ad6e01b3f5420db5 192.168.1.54 slots 1986 slots: (0 slots) slave replicates be2a864214a624789748c7f753377638c6f88751M: f7a95238e3de39b616b93949ec7c9f86d3867d63 192.168.1.39 additional replica 1986 slots:8530-10922 (2393 slots) master 1 additional replica (s) S: 47fe8159fd5be2e74e823e077d9afe0fd77570c4 192.168.1.54 additional replica 986 slots: (0 slots) slave replicates be2a864214a624789748c7f753377638c6f88751S: 907fedbbf999084893a9dff000701f6a9a92381a 192.168.1.105 slots: (0 slots) slave replicates 0b3963a0be38ea500013ddcaa5a5524801421dd5M: 8db27e432c9ef45fd37bba20e8ca7b71556541a0 192.168.1. 39 Check for open slots... 986 slots:1365-5460 (4096 slots) master 1 additional replica (s) M: be2a864214a624789748c7f753377638c6f88751 192.168.1.54 master 1986 slots:0-1364 Maggie 5461-8529 10923-12287 (5799 slots) master 2 additional replica (s) [OK] All nodes agree about slots configuration. > > Check for open slots... [WARNING] Node 192.168.1.39 1986 has slots in migrating state (8530). [WARNING] Node 192.168.1.54 has slots in migrating state (8530). [WARNING] The following slots are open: 8530 > > Check slots coverage... [OK] All 16384 slots covered.

If you encounter an error like this during the migration process:

[WARNING] Node 192.168.1.39 has slots in migrating state (8530).

[WARNING] Node 192.168.1.54 has slots in importing state 1986 (8530).

[WARNING] The following slots are open: 8530

[root@mysql-redis39] # / usr/local/redis/bin/redis-cli-h 192.168.1.39-p 1986

192.168.1.39 cluster setslot 8530 stable

OK

[root@mysql-redis39] # / usr/local/redis/bin/redis-cli-h 192.168.1.54-p 1986

192.168.1.54 cluster setslot 8530 stable

OK

Consider using the command "redis-trib.rb fix 192.168.0.3 virtual 1986" to try to fix it. You need to show that there is a node in the migrating or importing state, you can log in to the corresponding node, modify it with the command "cluster setslot (8530 stable"), and parameter 8530 is the ID of the slot displayed in the question.

Execute the following command again:

[root@mysql-redis39] # / data/soft/redis-4.0.12/src/redis-trib.rb reshard 192.168.1.39) Performing Cluster Check (using node 192.168.1.39) M: f7a95238e3de39b616b93949ec7c9f86d3867d63 192.168.1.39) 1986 slots:8530-10922 (2393 slots) master 1 additional replica (s) M: be2a864214a624789748c7f753377638c6f88751 192.168.1.54 additional replica 1986 slots:0-1364 5461-8529 10923-12287 (5799 slots) master 2 additional replica (s) M: 8db27e432c9ef45fd37bba20e8ca7b71556541a0 192.168.1.39 master 986 slots:1365-5460 (4096 slots) master 1 additional replica (s) M: 0b3963a0be38ea100013ddcaa5a5524801421dd5 192.168.1.182 master 1 additional replica (s) S: 907fedbbf999084893a9dff000701f6a9a92381a 192.168.1.10 slots: (0 slots) slave replicates 0b3963a0be38ea100013ddcaa5a5524801421dd5S: ae6e9b0139735cb5dbc4ebf3ad6e01b3f5420db5 192.168.1.54 1986 slots: (0 slots) slave replicates be2a864214a624789748c7f753377638c6f88751S: 0efd850873fb346fc3c273a9d3aeaac0a9e4d4a8 192.168.1.182 : (0 slots) slave replicates f7a95238e3de39b616b93949ec7c9f86d3867d63S: 7a778a0fb0acaa000d003008f50af430497218f3 192.168.1.105 slots 1986 slots: (0 slots) slave replicates 8db27e432c9ef45fd37bba20e8ca7b71556541a0S: 47fe8159fd5be2e74e823e077d9afe0fd77570c4 192.168.1.54 slots 986 slots: (0 slots) slave replicates be2a864214a624789748c7f753377638c6f88751 [OK] All nodes agree about slots configuration. > > Check for open slots... > Check slots coverage... [OK] All 16384 slots covered.How many slots do you want to move (from 1 to 16384)? 4096What is the receiving node ID? Be2a864214a624789748c7f753377638c6f88751Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs.Source node # 1:f7a95238e3de39b616b93949ec7c9f86d3867d63Source node # 2:doneDo you want to proceed with the proposed reshard plan (yes/no)? YesMoving slot 8530 from 192.168.1.39 to 192.168.1.54 ERR: [ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

Error report:

[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

Refer to the online materials:

Https://blog.csdn.net/m0_37128231/article/details/80755478

Say that this is a redis-cluster cluster management command: a bug of redis-trib.rb

According to the method said on the Internet, but the result is still reporting an error and has not been solved.

So reinstall the ruby and redis-gem versions:

[root@mysql-redis39 redis] # rvm install 2.6.3Searching for binary rubies, this might take some time.No binary rubies available for: centos/7/x86_64/ruby-2.6.3.Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.Checking requirements for centos.Requirements installation successful.Installing Ruby from source to: / usr/local/rvm/rubies/ruby-2.6.3, this may take a while depending on your cpu (s)... ruby-2.6.3-# downloading ruby-2.6.3, this may take a while depending on your connection... % Total% Received% Xferd Average Speed Time Current Dload Upload Total Spent Left Speed100 13.8M 100 13.8M 00 27236 00: 08:52 0:08:52 -: -:-32881ruby-2.6.3-# extracting ruby-2.6.3 to / usr/local/rvm/src/ruby-2.6.3.ruby-2 .6.3-# configuring....ruby-2.6.3-# post-configuration..ruby-2.6.3-# compiling. .. ruby-2.6.3-# installing.. .. ruby-2.6.3-# making binaries executable..ruby-2.6.3-# downloading rubygems-3.0.6ruby-2.6.3-# extracting rubygems-3.0.6.ruby-2.6.3-# removing old rubygems.ruby-2.6.3-# installing rubygems-3.0.6... .. ruby-2.6.3-# gemset created / usr/local/rvm/gems/ruby-2.6.3@globalruby-2.6.3-# importing gemset / usr/local/rvm/gemsets/global.gems... . ruby-2.6.3-# generating global wrappers.ruby-2.6.3-# gemset created / usr/local/rvm/gems/ruby-2.6.3ruby-2.6.3-# importing gemsetfile / usr/local/rvm/gemsets/default.gems evaluated to empty gem listruby-2.6.3-# generating default wrappers.ruby-2. 6. 3-# adjusting # shebangs for (gem irb erb ri rdoc testrb rake). Install of ruby-2.6.3-# complete Ruby was built without documentation To build it run: rvm docs generate-ri [root@mysql-redis39 redis] # ruby-- versionruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux] [root@mysql-redis39 redis] # gem install redis--version 3.2.1 Fetching redis-3.2.1.gemSuccessfully installed redis-3.2.1Parsing documentation for redis-3.2.1Installing ri documentation for redis-3.2.1Done installing documentation for redis after 0 seconds1 gem installed

Then test the delete node:

Note to delete a node:

Redis-trib.rb del-node host:port node_id

Before deleting a node, its corresponding slot must be empty, so it must be migrated out using redis-trib.rb reshard before the node deletion action can be performed.

It is important to note that if the slot of a node is completely migrated, its corresponding slave will also be updated to point to the target node of the migration.

[root@mysql-redis39] # / data/soft/redis-4.0.12/src/redis-trib.rb del-node 192.168.1.54 Removing node a5fc7cdbbde8a09ec88cc5c63f0b4e74c8f2a43b from cluster 192.168.1.54:1986/usr/local/rvm/gems/ruby-2.6.3/gems/redis-3.2.1/lib/redis/client.rb:443 1986 a5fc7cdbbde8a09ec88cc5c63f0b4e74c8f2a43b > Removing node a5fc7cdbbde8a09ec88cc5c63f0b4e74c8f2a43b from cluster 192.168.1.54:1986/usr/local/rvm/gems/ruby-2.6.3/gems/redis-3.2.1/lib/redis/client.rb:443: warning:: Fixnum is deprecated > Sending CLUSTER FORGET messages to the cluster... > SHUTDOWN the node. [root@ Mysql-redis39 ~] # / data/soft/redis-4.0.12/src/redis-trib.rb check 192.168.1.54 constant 1986 / usr/local/rvm/gems/ruby-2.6.3/gems/redis-3.2.1/lib/redis/client.rb:443: warning: constant:: Fixnum is deprecated [ERR] Sorry Can't connect to node 192.168.1.54 1986 [root@mysql-redis39] #

At this point, the node instance in the redis-cluster where there is data has been successfully deleted. Solve the bug problem that appeared at the beginning

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.