Shulou(Shulou.com)06/01 Report--

This article mainly explains how to mount a new disk without restart in HDFS. Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Next let the editor to take you to learn "HDFS restart-free mount new disk method" it!

In the production environment, the disk size of the cluster node varies with its utilization. Although HDFS has a balancing strategy, there will be data imbalance. Some node disks will be full, and then this node will be unhealthy (Unhealthy Nodes). Yarn's disk threshold check (yarn.nodemanager.disk-health-checker.min-healthy-disks) defaults to 90%. If it exceeds this value, it will be unhealthy. A node in the cluster will be unhealthy. It will cause the task to run slowly, then the task will be squeezed, and the Yarn resources will be centrally occupied, which will eventually affect the upper-level services.

problem

Insufficient disk space, to be expanded, disks can be dynamically added and deleted

Insufficient yarn resources, need to be optimized, and can be dynamically adjusted

Official statement

Official document: DataNode Hot Swap Drive

DataNode Hot Swap Drive

Datanode supports hot swappable drives. The user can add or replace HDFS data volumes without shutting down the DataNode. The following briefly describes the typical hot swapping drive procedure:

If there are new storage directories, the user should format them and mount them appropriately.

The user updates the DataNode configuration dfs.datanode.data.dir to reflect the data volume directories that will be actively in use.

The user runs dfsadmin-reconfig datanode HOST:PORT start to start the reconfiguration process. The user can use dfsadmin-reconfig datanode HOST:PORT status to query the running status of the reconfiguration task.

Once the reconfiguration task has completed, the user can safely umount the removed data volume directories and physically remove the disks.

Translated by Google as follows:

DataNode hot-swappable drive

Datanode supports hot-swappable drives. Users can add or replace HDFS data volumes without shutting down DataNode. The following is a brief description of a typical hot-swappable drive process:

If there are new storage directories, the user should format them and mount them appropriately.

The user updates the DataNode configuration dfs.datanode.data.dir to reflect the volume directory that will be actively used.

The user runs dfsadmin-reconfig datanode HOST:PORT start to start the reconfiguration process. Users can use dfsadmin-reconfig datanode HOST:PORT status to query the running status of the reconfiguration task.

After the reconfiguration task is complete, the user can safely unmount the removed data volume directory and physically remove the disk.

We first talk about adding a new disk, the general idea is that there is no need to restart any node, on the node that needs to add a disk, add a good disk, this requires operation and maintenance operation, that is, mount a new directory, the document says to configure the DataNode directory, here we suggest to add the NameNode directory dfs.namenode.name.dir, and then execute a command to make the configuration effective, and finally check the configuration status.

Concrete operation

Here, it is assumed that there is an existing cluster, and the node that needs to add disk is node2. The operation and maintenance staff have also helped us mount the disk, and the new disk directory is / data2.

Step 1: add a new directory

Add new directories, data directories, and metadata directories to node2

Mkdir-p / data2/soft/hadoop/tmp/dfs/data

Mkdir-p / data2/soft/hadoop/tmp/dfs/name

Step 2: modify the configuration

Modify the configuration file hdfs-site.xml of node node2. There are two main attributes

Before modification

Dfs.namenode.name.dir / data/soft/hadoop/tmp/dfs/name dfs.datanode.data.dir / data/soft/hadoop/tmp/dfs/data

After modification

Dfs.namenode.name.dir / data/soft/hadoop/tmp/dfs/name,/data2/soft/hadoop/tmp/dfs/name dfs.datanode.data.dir / data/soft/hadoop/tmp/dfs/data,/data2/soft/hadoop/tmp/dfs/data step 3: make the configuration effective

Reconfigure the configuration of DataNode so that the configuration takes effect.

Hdfs dfsadmin-reconfig datanode node2:50020 start

Hdfs dfsadmin-reconfig datanode node2:50020 status

Step 4: balance the data

HDFS data itself is not always evenly placed in DataNode. After adding new disks, we need to rebalance the data. HDFS provides us with a tool to analyze the location of data blocks and rebalance data across DataNode: balancer

Official document: hdfs balancer

Hdfs balancer [- threshold]: the disk capacity is 100%, which is the target parameter to determine whether the cluster is balanced. The difference between each datanode storage utilization and the total cluster storage utilization should be less than this threshold. The smaller the storage utilization, the more balanced it is. The default is 10, and the larger the balance is, the faster the balance is. [- policy]: datanode (default): if each data node is balanced, the cluster is balanced. Blockpool: if each block pool in each data node is balanced, the cluster is balanced. [- exclude [- f |]]: excludes the specified data node from the balance of the balancer. [- include [- f |]]: includes only the specified data nodes to be balanced by the balancer. [- idleiterations]: the number of threads started by rebalanecing server. Default is 5. Pre-balance operation

The maximum number of bytes per second used by each data node is limited due to the influence of bandwidth during balance, so we will set this value temporarily. This value will override the value of dfs.datanode.balance.bandwidthPerSec in hdfs-site.xml. The default value is 1m. This operation command is not persistent, as shown below:

Hdfs dfsadmin-setBalancerBandwidth 104857600

Other values: 1024 million 1024 million 1m (1048576), 52428800 million 50M people 104857600 million 100m

Begin to balance

Execute the balance command this time. The number of disk balance targets is 20 nohup hdfs balancer-threshold 20 > balancer.log &.

Other balance commands

Balance all nodes nohup hdfs balancer > balancer.log &

Balance the specified node. The target number of disk balance is 10 nohup hdfs balancer-include node7,node9,node10,node11-threshold 10 > balancer.log &

Balance the specified node, disk balance target number, start threads 10 nohup hdfs balancer-include node7,node9,node10,node11-threshold 10-idleiterations 10 > balancer.log &

Note (optional): generally, when balancing, you can stop storing NodeManager on a higher node, so that it will not be affected by the local NodeManager falling data to the local node, resulting in a rapid increase in local storage.

Yarn-daemon.sh stop nodemanager yarn-daemon.sh start nodemanager

At this point, I believe you have a deeper understanding of the "HDFS restart-free method of mounting a new disk". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.