Shulou(Shulou.com)06/03 Report--

Monday, 2019-1-21

2.3.1. Balancer

Run a cluster balancing tool. The administrator can stop the rebalancing process by pressing Ctrl+ C.

The need for data balance

The reasons for the imbalance of HDFS data

1. The hard disk storage in a DataNode machine has reached its saturation value.

two。 Add or delete nodes in the cluster.

The impact of data imbalance

1.Map tasks may be assigned to machines that do not store data, resulting in a failure to implement local computing, which eventually leads to the network

Bandwidth consumption.

two。 When some data nodes are fully loaded, the new data blocks will only be stored on the node machines with spare data, resulting in the possibility of parallel reading.

Requirements of the data balancing process

1. Data balancing does not result in block reduction and loss of block backups.

two。 The administrator can abort the data balancing process.

3. The size of each block movement should be controllable so that a blocking network can be placed.

4.namenode will not be too busy because of the data balancing service.

Principle of automatic data balance

Due to the use of balancing algorithm, data balancing is an iterative and recurring process. The ultimate goal of each iteration

Is to allow high-load machines to reduce the data load, so data balancing will maximize the use of network bandwidth.

Data balance process interaction diagram

The steps are analyzed as follows:

1. Data equalization service (Rebalancing Server) first requires NameNode to generate DataNode data distribution analysis report to obtain the usage of each DataNode disk. 2. Rebalancing Server summarizes the data distribution to be moved, and calculates the specific data block migration roadmap. Block migration roadmap, ensure the shortest path within the network 3, start the block migration task, Proxy Source DataNode copy a block to move 4, copy the replicated block to the target DataNode 5, delete the original block 6, the target DataNode confirms the completion of the block migration to the Proxy Source DataNode 7, Proxy Source DataNode confirms the completion of the block migration to the Rebalancing Server. Then continue the process until the cluster reaches the standard of data equalization

Actual operation

1. Switch to hdfs user

[root@hadoop-master] # su-hdfs2. Check the current data distribution [hdfs@hadoop-master ~] $hdfs dfsadmin-report > / tmp/bq [hdfs@hadoop-master ~] $cat / tmp/bq Configured Capacity: 273287419086 (254.52 GB) Present Capacity: 209643254756 (195.25 GB) DFS Remaining: 199579415524 (185.87 GB) DFS Used: 10063839232 (9.37 GB) DFS Used%: 4.80%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0Missing blocks (with replication factor 1): 0 configuration capacity: 273287419086 ( 254.52 GB) existing capacity: 209643254756 (195.25 GB) DFS remaining: 199579415524 (185.87 GB) using DFS:10063839232 (9.37 GB) using DFS%:4.80% under the replica block: 0 blocks with damaged copies: 0 missing blocks: 0 missing blocks (replication factor 1): 0 color- -Live datanodes (3): real-time data node (3): Name: 192.168.0.117hadoop-node01Rack 50010 (hadoop-node01) Hostname: hadoop-node01Rack: / defaultDecommission Status: NormalConfigured Capacity: 91095806362 (84.84 GB) DFS Used: 3354603520 (3.12 GB) Non DFS Used: 12246245786 (11.41 GB) DFS Remaining: 69809631564 (65.02 GB) DFS Used%: 3.68%DFS Remaining%: 76.63%Configured Cache Capacity: 4294967296 (4) GB) Cache Used: 0 (0 B) Cache Remaining: 4294967296 (4 GB) Cache Used%: 0.00%Cache Remaining%: 100.00%Xceivers: 10Last contact: Mon Jan 21 10:45:24 CST 2019 name: 192.168.0.117 Cache Remaining 50010 (hadoop-node01) hostname: hadoop- Node01 Rack: / default decommissioning status: normal configuration capacity: 91095806362 (84.84 GB) for DFS:3354603520 (3.12 GB) non-DFS use: 12246245786 (11.41) GB) remaining DFS:69809631564 (65.02 GB) using DFS%:3.68%DFS remaining%: 76.63% configured cache capacity: 4294967296 (4 GB) used cache: 0 (0 B) remaining cache: 4294967296 (4 GB) used cache%: 0.005% remaining cache%: 100.00%Xceivers:10 Last contact: Mon Jan 21 10:45:24 CST 2019Name: 192.168.0.11818 DFS%:3.68%DFS 50010 ( Hadoop-master) Hostname: hadoop-masterRack: / defaultDecommission Status: NormalConfigured Capacity: 91095806362 (GB) DFS Used: 3354632192 (3.12 GB) Non DFS Used: 29517959578 (27.49 GB) DFS Remaining: 52537889100 (48.93 GB) DFS Used%: 3.68%DFS Remaining%: 57.67%Configured Cache Capacity: 4294967296 (4 GB) Cache Used: 0 (0B) Cache Remaining: 4294967296 (4 GB) Cache Used%: 0.00%Cache Remaining%: 100.00%Xceivers: 10Last contact: Mon Jan 21 10:45:24 CST 2019Name: 192.168.0.121 CST 2019Name 50010 (hadoop-node02) Hostname: hadoop-node02Rack: / defaultDecommission Status: NormalConfigured Capacity: 91095806362 (84.84 GB) DFS Used: 3354603520 (3.12 GB) Non DFS Used: 4823982490 (4.49 GB) DFS Remaining: 77231894860 (71.93 GB) DFS Used%: 3.68%DFS Remaining%: 84.78%Configured Cache Capacity: 4294967296 (4 GB) Cache Used: 0 B) Cache Remaining: 4294967296 (4 GB) Cache Used%: 0.00%Cache Remaining%: 100.00%Xceivers: 10Last contact: Mon Jan 21 10:45:24 CST 20193. Use the command to balance data [hdfs@hadoop-master ~] $hdfs balancer19/01/21 10:49:19 INFO balancer.Balancer: namenodes = [hdfs://vg-cdh-test] 19-01-21 10:49:19 INFO balancer.Balancer: parameters = Balancer.Parameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, number of nodes to be excluded = 0, number of nodes to be included = 0 Run during upgrade = false] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved19/01/21 10:49:21 INFO net.NetworkTopology: Adding a new node: / default/192.168.0.117:5001019/01/21 10:49:21 INFO net.NetworkTopology: Adding a new node: / default/192.168.0.118:5001019/01/21 10:49:21 INFO net.NetworkTopology: Adding a new node: / default/192.168.0 .121: 5001019/01/21 10:49:21 INFO balancer.Balancer: 0 over-utilized: [] 19-01-21 10:49:21 INFO balancer.Balancer: 0 underutilized: [] The cluster is balanced. Exiting...2019-1-21 10:49:21 00 B 0 B-1 B2019-1-21 10:49:21 Balancing took 2.738 seconds4. Check the data distribution after data balance [hdfs@hadoop-master ~] $hdfs dfsadmin-report > / tmp/bh [hdfs@hadoop-master ~] $cat / tmp/bhConfigured Capacity: 273287419086 (254.52 GB) Present Capacity: 209660106924 (195.26 GB) DFS Remaining: 199596266468 (185.89 GB) DFS Used: 10063840456 (9.37 GB) DFS Used%: 4.80%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0Missing blocks (with replication factor 1): 0- -Live datanodes (3): Name: 192.168.0.117 Hostname 50010 (hadoop-node01) Hostname: hadoop-node01Rack: / defaultDecommission Status: NormalConfigured Capacity: 91095806362 (84.84 GB) DFS Used: 3354603928 (3.12 GB) Non DFS Used: 12246663170 (11.41 GB) DFS Remaining: 69809213772 (65.01 GB) DFS Used %: 3.68%DFS Remaining%: 76.63%Configured Cache Capacity: 4294967296 (4 GB) Cache Used: 0 (0 B) Cache Remaining: 4294967296 (4 GB) Cache Used%: 0.00%Cache Remaining%: 100.00%Xceivers: 10Last contact: Mon Jan 21 10:50:10 CST 2019Name: 192.168.0.118 hadoop-master 50010 (hadoop-master) Hostname: hadoop-masterRack: / defaultDecommission Status: NormalConfigured Capacity: 91095806362 (84.84 GB) DFS Used: 3354632600 (3.12 GB) Non DFS Used: 2950144428748 (48.95 GB) DFS Used%: 3.68%DFS Remaining%: 57.69%Configured Cache Capacity: 4294967296 (4 GB) Cache Used: 0 (0 B) Cache Remaining: 4294967296 (4 GB) Cache Used%: 0.00%Cache Remaining%: 100.00%Xceivers: 10Last contact: Mon Jan 21 10:50:10 CST 2019Name: 192.168.0.121 DFS Used%: 3.68%DFS Remaining%: 57.69%Configured Cache Capacity 50010 (hadoop-node02) Hostname: hadoop-node02Rack: / defaultDecommission Status: NormalConfigured Capacity: 91095806362 ( 84.84 GB) DFS Used: 3354603928 (3.12 GB) Non DFS Used: 4823252994 (4.49 GB) DFS Remaining: 77232623948 (71.93 GB) DFS Used%: 3.68%DFS Remaining%: 84.78%Configured Cache Capacity: 4294967296 (4 GB) Cache Used: 0 (0B) Cache Remaining: 4294967296 (4 GB) Cache Used%: 0.00%Cache Remaining%: 100.00%Xceivers: 10Last contact: Mon Jan 21 10:50:10 CST 2019- -5. Compare the report information before and after data balance [hdfs@hadoop-master ~] $diff / tmp/bq / tmp/bh2,4c2,4

< Present Capacity: 209643254756 (195.25 GB)< DFS Remaining: 199579415524 (185.87 GB)< DFS Used: 10063839232 (9.37 GB)--->

Present Capacity: 209660106924 (195.26 GB) > DFS Remaining: 199596266468 (185.89 GB) > DFS Used: 10063840456 (9.37 GB)

< DFS Used: 3354603520 (3.12 GB)< Non DFS Used: 12246245786 (11.41 GB)< DFS Remaining: 69809631564 (65.02 GB)--->

DFS Used: 3354603928 (3.12 GB) > Non DFS Used: 12246663170 (11.41 GB) > DFS Remaining: 69809213772 (65.01 GB) 30c30

< Last contact: Mon Jan 21 10:45:24 CST 2019--->

Last contact: Mon Jan 21 10:50:10 CST 201938,40c38,40

< DFS Used: 3354632192 (3.12 GB)< Non DFS Used: 29517959578 (27.49 GB)< DFS Remaining: 52537889100 (48.93 GB)--->

DFS Used: 3354632600 (3.12 GB) > Non DFS Used: 29501419522 (27.48 GB) > DFS Remaining: 52554428748 (48.95 GB) 42c42

< DFS Remaining%: 57.67%--->

DFS Remaining%: 57.69%49c49

< Last contact: Mon Jan 21 10:45:24 CST 2019--->

Last contact: Mon Jan 21 10:50:10 CST 201957,59c57,59

< DFS Used: 3354603520 (3.12 GB)< Non DFS Used: 4823982490 (4.49 GB)< DFS Remaining: 77231894860 (71.93 GB)--->

DFS Used: 3354603928 (3.12 GB) > Non DFS Used: 4823252994 (4.49 GB) > DFS Remaining: 77232623948 (71.93 GB) 68c68

< Last contact: Mon Jan 21 10:45:24 CST 2019--->

Last contact: Mon Jan 21 10:50:10 CST 2019 the actual operation in production is as follows: hdfs dfsadmin-fs hdfs://uhadoop-mzwc2w-master2:8020-setBalancerBandwidth 3145728000 [hadoop@uhadoop-mzwc2w-master1 ~] $hdfs dfsadmin-fs hdfs://uhadoop-mzwc2w-master2:8020-setBalancerBandwidth 3145728000Balancer bandwidth is set to 3145728000 [hadoop@uhadoop-mzwc2w-master1 ~] $hdfs dfsadmin-fs hdfs://uhadoop-mzwc2w-master1:8020-setBalancerBandwidth 3145728000Balancer bandwidth is set to 3145728000 / must have 2 nodes Namenode node master1 master2 is executed in production on Monday, 2019-7-22. I executed balancer [hadoop@uhadoop-mzwc2w-master1 majihui0718] $nohup hdfs balancer > balancer.log & / / on both master1 and master2. [hadoop@uhadoop-mzwc2w-master2 majihui0722] $pwd/home/hadoop/majihui0722 [hadoop@uhadoop-mzwc2w-master2 majihui0722] $lltotal 596 RWMui Rwlashi-1 hadoop wheel 609794 Jul 22 14:11 balancer.log We are 1000m networks Give 300m of bandwidth for datanode's data balance

Reference link

Https://www.cnblogs.com/qingyunzong/p/8535995.html

Reference link

Detailed explanation of HDFS balance Strategy https://www.jianshu.com/p/f7c1cd476601

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.