Shulou(Shulou.com)06/02 Report--

HDFS is only one of the most basic services of Hadoop, and many other services are based on HDFS. Therefore, the deployment of a HDFS cluster is not only a very core action, but also the beginning of big data platform.

To install a Hadoop cluster, you need to have Zookeeper before you can complete the installation. If there is no Zookeeper, please deploy a set of Zookeeper first. In addition, JDK and some settings of the physical host, etc. Please refer to:

Hadoop Cluster (1) Zookeeper Construction

Hadoop Cluster (3) Hbase Construction

Hadoop Cluster (4) Hadoop upgrade

Let's start the installation of HDFS

HDFS host allocation

123192.168.67.101 c6701-Namenode+datanode192.168.67.102 c6702-datanode192.168.67.103 c6703-datanode

1. Install HDFS and extract hadoop-2.6.0-EDH-0u2.tar.gz

I download both 2.6 and 2.7 versions of the software, install 2.6 first, and then perform the upgrade steps from 2.6 to 2.7

Useradd hdfsecho "hdfs:hdfs" | chpasswdsu-hdfscd / tmp/softwaretar-zxvf hadoop-2.6.0-EDH-0u2.tar.gz-C / home/hdfs/mkdir-p / data/hadoop/temp mkdir-p / data/hadoop/journal mkdir-p / data/hadoop/hdfs/name mkdir-p / data/hadoop/hdfs/datachown-R hdfs:hdfs / data/hadoopchown-R hdfs:hdfs / data/hadoop/temp chown-R hdfs:hdfs / data/hadoop/journal chown-R hdfs:hdfs / data/ Hadoop/hdfs/name chown-R hdfs:hdfs / data/hadoop/hdfs/data $pwd/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop

two。 Modify the parameters corresponding to core-site.xml

$cat core-site.xml fs.defaultFS hdfs://ns hadoop.tmp.dir / data/hadoop/temp io.file.buffer.size 4096 ha.zookeeper.quorum c6701 ha.zookeeper.quorum 2181 Magazine c6702 purl 2181

3. Modify the parameters corresponding to hdfs-site.xml

Cat hdfs-site.xml dfs.nameservices ns dfs.ha.namenodes.ns nn1 Nn2 dfs.namenode.rpc-address.ns.nn1 c6701:9000 dfs.namenode.http-address.ns.nn1 c6701:50070 dfs.namenode.rpc-address.ns.nn2 c6702:9000 dfs.namenode.http-address.ns.nn2 c6702:50070 dfs.namenode.shared.edits.dir qjournal://c6701:8485 C6702:8485 C6703:8485/ns dfs.journalnode.edits.dir / data/hadoop/journal dfs.ha.automatic-failover.enabled true dfs.client.failover.proxy.provider.ns org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.fencing.methods Sshfence dfs.ha.fencing.ssh.private-key-files / home/hdfs/.ssh/id_rsa dfs.namenode.name.dir / data/hadoop/hdfs/name dfs.datanode.data.dir / data/hadoop/hdfs/data Dfs.replication 2 dfs.webhdfs.enabled true

4. Add a slaves file

$more slavesc6701c6702c6703

-install hdfs--- for C6702

5. Create c6702 users and ssh secret-free for hdfs users

Ssh c6702 "useradd hdfs" ssh c6702 "echo" hdfs:hdfs "| chpasswd" ssh-copy-id hdfs@c6702

6. Copy softwar

Scp-r / tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6702:/tmp/software/.ssh c6702 "chmod 777 / tmp/software/*"

7. Create a directory and extract the software

Ssh hdfs@c6702 "mkdir hdfs" ssh hdfs@c6702 "tar-zxvf / tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz-C / home/hdfs" ssh hdfs@c6702 "ls-al hdfs" ssh hdfs@c6702 "ls-al hdfs/hadoop*"

Copy Profil

Ssh hdfs@c6702 "rm-rf / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml" ssh hdfs@c6702 "rm-rf / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml" scp-r / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/ Etc/hadoop/core-site.xmlscp-r / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xmlscp-r / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves

Create the directories required by hdfs

Ssh root@c6702 "mkdir-p / data/hadoop" ssh root@c6702 "chown-R hdfs:hdfs / data/hadoop" ssh hdfs@c6702 "mkdir-p / data/hadoop/temp" ssh hdfs@c6702 "mkdir-p / data/hadoop/journal" ssh hdfs@c6702 "mkdir-p / data/hadoop/hdfs/name" ssh hdfs@c6702 "mkdir-p / data/hadoop/hdfs/data"

-install hdfs--- for C6703

8. Create c6703 users and ssh secret-free for hdfs users

Ssh c6703 "useradd hdfs" ssh c6703 "echo" hdfs:hdfs "| chpasswd" ssh-copy-id hdfs@c6703

9. Copy softwar

Scp-r / tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6703:/tmp/software/.ssh c6703 "chmod 777 / tmp/software/*" 10. Create a directory and extract the software ssh hdfs@c6703 "mkdir hdfs" ssh hdfs@c6703 "tar-zxvf / tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz-C / home/hdfs" ssh hdfs@c6703 "ls-al hdfs" ssh hdfs@c6703 "ls-al hdfs/hadoop*"

Copy Profil

Ssh hdfs@c6703 "rm-rf / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml" ssh hdfs@c6703 "rm-rf / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml" scp-r / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/ Etc/hadoop/core-site.xmlscp-r / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xmlscp-r / home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves

Create the directories required by hdfs

Ssh root@c6703 "mkdir-p / data/hadoop" ssh root@c6703 "chown-R hdfs:hdfs / data/hadoop" ssh hdfs@c6703 "mkdir-p / data/hadoop/temp" ssh hdfs@c6703 "mkdir-p / data/hadoop/journal" ssh hdfs@c6703 "mkdir-p / data/hadoop/hdfs/name" ssh hdfs@c6703 "mkdir-p / data/hadoop/hdfs/data"

11. To start HDFS, start the journalnode of three nodes first

/ home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start journalnode

Check statu

$jps3958 Jps3868 JournalNode

twelve。 Then start namenode. Before starting namenode for the first time, format namenode information in one of the nodes (master node). The information will exist in the path specified by dfs.namenode.name.dir.

Dfs.namenode.name.dir / data/hadoop/hdfs/name$. / hdfs namenode-format17/09/26 07:52:17 INFO namenode.NameNode: STARTUP_MSG:/***STARTUP_MSG: Starting NameNodeSTARTUP_MSG: host = c6701.python279.org/ 192.168.67.101STARTUP_MSG: args = [- format] STARTUP_MSG: version = 2.6.0-EDH-0u2STARTUP_MSG: classpath = / home/hdfs/hadoop-2.6.0-EDHxxxxxxxxxxSTARTUP_MSG: build = http://gitlab-xxxxxSTARTUP_MSG: java = 1.8.0 "144" * * / 07:52:17 on 17-09-26 INFO namenode.NameNode: registered UNIX signal handlers for [TERM HUP, INT] 07:52:17 on 17-09-26 INFO namenode.NameNode: createNameNode [- format] 17-09-26 07:52:18 WARN common.Util: Path / data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.17/09/26 07:52:18 WARN common.Util: Path / data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.Formatting using clusterid: CID-b2f01411-862f-44b2-a6dc-7d17bd48d52217/09/26 07:52:18 INFO namenode.FSNamesystem: No KeyProvider found.17/09/26 07:52:18 INFO namenode.FSNamesystem: fsLock is fair:true17/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=100017/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true17/09/26 07:52:18 INFO blockmanagement.BlockManager: Dfs.namenode.startup.delay.block.deletion.sec is set to 000000 Sep 26 07:52:18 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Sep 26 07 The block deletion will start around 52 1817 Grammer 26 07:52:18 INFO util.GSet: Computing capacity for map BlocksMap17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit17/09/26 07:52:18 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB17/09/26 07:52:18 INFO util.GSet: capacity = 2 ^ 21 = 2097152 entries17/09/26 07:52:18 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false17/09/26 07:52:18 INFO blockmanagement.BlockManager: defaultReplication = 217-09-26 07:52:18 INFO blockmanagement.BlockManager: maxReplication = 51217 Raymond 09ax 26 07:52:18 INFO blockmanagement.BlockManager: minReplication = 117-09-26 07:52:18 INFO blockmanagement.BlockManager: maxReplicationStreams = 217-09-26 07:52:18 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false17/09/26 07:52:18 INFO blockmanagement.BlockManager: replicationRecheckInterval = 300017 INFO blockmanagement.BlockManager: encryptDataTransfer = false17/09/26 07:52:18 07:52:18 INFO blockmanagement.BlockManager: encryptDataTransfer = false17/09/26 07:52:18 INFO namenode.FSNamesystem: fsOwner = hdfs (auth:SIMPLE) 17 / 09 take 26 07:52:18 INFO namenode.FSNamesystem: supergroup = supergroup17/09/26 07:52:18 INFO namenode.FSNamesystem: isPermissionEnabled = true17/09/26 07:52:18 INFO namenode.FSNamesystem: Determined nameservice ID: ns17/09/26 07:52:18 INFO namenode.FSNamesystem: HA Enabled: true17/09/26 07:52:18 INFO namenode.FSNamesystem: Append Enabled: true17/09/26 07:52:18 INFO util.GSet: Computing capacity for map INodeMap17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit17/09/26 07:52:18 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB17/09/26 07:52:18 INFO util.GSet: capacity = 2 ^ 20 = 1048576 entries17/09/26 07:52:18 INFO namenode.NameNode: Caching file names occuring more than 10 times17/09/26 07:52:18 INFO util.GSet: Computing capacity for map cachedBlocks17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit17/09/26 07:52:18 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB17/09/26 07:52:18 INFO util.GSet: capacity = 2 ^ 18 = 262144 entries17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.999000012874603317 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 017 INFO namenode.FSNamesystem 26 07:52: 18 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 3000017 INFO namenode.FSNamesystem 26 07:52:18 INFO namenode.FSNamesystem: Retry cache on namenode is enabled17/09/26 07:52:18 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis17/09/26 07:52:18 INFO util.GSet: Computing capacity for map NameNodeRetryCache17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit17/09/26 07:52: 18 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB17/09/26 07:52:18 INFO util.GSet: capacity = 2 ^ 15 = 32768 entries17/09/26 07:52:18 INFO namenode.NNConf: ACLs enabled? False17/09/26 07:52:18 INFO namenode.NNConf: XAttrs enabled? True17/09/26 07:52:18 INFO namenode.NNConf: Maximum size of an xattr: 1638417 INFO namenode.FSImage 09 INFO namenode.FSImage 26 07:52:19 INFO namenode.FSImage: Allocated new BlockPoolId: BP-144216011-192.168.67.101-150641233975717 INFO common.Storage 26 07:52:19 INFO common.Storage: Storage directory / data/hadoop/hdfs/name has been successfully formatted.17/09/26 07:52:20 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid > = 017 INFO namenode.NNStorageRetentionManager 09 INFO util.ExitUtil 26 07:52:20 INFO util.ExitUtil: Exiting with status 017Accord 09 INFO namenode.NameNode 26 07:52:20 INFO namenode.NameNode: SHUTDOWN_MSG:/***SHUTDOWN_MSG: Shutting down NameNode at c6701.python279.orgAccord 192.168.67.101 * * * /

13. Standby namenode needs to execute bootstrapstandby first, and the output is as follows

[hdfs@c6702 sbin] $.. / bin/hdfs namenode-bootstrapstandby17/09/26 09:44:58 INFO namenode.NameNode: STARTUP_MSG:/***STARTUP_MSG: Starting NameNodeSTARTUP_MSG: host = c6702.python279.org/192.168.67.102STARTUP _ MSG: args = [- bootstrapstandby] STARTUP_MSG: version = 2.6.0-EDH-0u2STARTUP_MSG: classpath = / home/hdfs/haxxxSTARTUP_MSG: build = http://gitlab-xxxxSTARTUP_MSG: java = 1.8.0 "144mm * * / 09:44:58 on 17-09-26 INFO namenode.NameNode: registered UNIX signal handlers for [TERM HUP, INT] 09:44:58 on 17-09-26 INFO namenode.NameNode: createNameNode [- bootstrapstandby] 17-09-26 09:44:59 WARN common.Util: Path / data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.17/09/26 09:44:59 WARN common.Util: Path / data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.===About to bootstrap Standby ID nn2 from: Nameservice ID: ns Other Namenode ID: nn1 Other NN's HTTP address: http://c6701:50070 Other NN's IPC address: c6701 pick 192.168.67.101 ns Other Namenode ID 9000 Namespace ID: 793662207 Block pool ID: BP-144216011-192.168.67.101-1506412339757 Cluster ID: CID-b2f01411-862f-44b2-a6dc-7d17bd48d522 Layout version:-60===Re-format filesystem in Storage Directory / data/hadoop/hdfs/name? (Y or N) y17 INFO common.Storage 09 INFO common.Storage 26 09:45:16: Storage directory / data/hadoop/hdfs/name has been successfully formatted.17/09/26 09:45:16 WARN common.Util: Path / data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.17/09/26 09:45:16 WARN common.Util: Path / data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.17/09/26 09:45:17 INFO namenode.TransferFsImage: Opening connection to http://c6701:50070/imagetransfer?getimage=1&txid=0&storageInfo=-60:793662207:0:CID-b2f01411-862f-44b2-a6dc-7d17bd48d52217/09/26 09:45:17 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds17/09/26 09:45:17 INFO namenode.TransferFsImage: Transfer took 0.01s at 0.00 KB/s17/09/26 09:45:17 INFO namenode.TransferFsImage: Downloaded file fsimage .ckpt _ 0000000000000000000 size 351 bytes.17/09/26 09:45:17 INFO util.ExitUtil: Exiting with status 017/09/26 09:45:17 INFO namenode.NameNode: SHUTDOWN_MSG:/***SHUTDOWN_MSG: Shutting down NameNode at c6702.python279.org/ 192.168.67.102 million dollars /

14. Check the status. Namenode hasn't started yet.

[hdfs@c6702 sbin] $jps4539 Jps3868 JournalNode

15. Start standby namenode, command and master start in the same way

[hdfs@c6702 sbin] $. / hadoop-daemon.sh start namenodestarting namenode, logging to / home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hdfs-namenode-c6702.python279.org.out

16. Check again, namenode has been started

[hdfs@c6702 sbin] $jps4640 Jps4570 NameNode3868 JournalNode

17. Format zkfc, let the ha node be generated in zookeeper, and execute the following command on master to complete the formatting

[hdfs@c6701 bin] $. / hdfs zkfc-formatZK17/09/26 09:59:20 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at c6701 INFO tools.DFSZKFailoverController: 192.168.67.101 formatZK17/09/26 900017 Grey 09 Grey 26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965 Built on 02/20/2014 09:09 GMT17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:host.name=c6701.python279.org17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_14417/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/jdk1.8.0_144/jre17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/exxxx17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/lib/native17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java INFO zookeeper.ZooKeeper: Client environment:os.name=Linux17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd6417/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_6417/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hdfs17/09 / 26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfs/hadoop-2.6.0-EDH-0u2/bin17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Initiating client connection ConnectString=c6701:2181,c6702:2181,c6703:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@20deea7f17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Opening socket connection to server c6703.python279.org/192.168.67.103:2181. Will not attempt to authenticate using SASL (unknown error) 09:59:20 on 17-09-26 INFO zookeeper.ClientCnxn: Socket connection established to c6703.python279.org/192.168.67.103:2181, initiating session17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Session establishment complete on server c6703.python279.org/192.168.67.103:2181, sessionid = 0x35ebc5163710000 Negotiated timeout = 500017 0x35ebc5163710000 closed17/09/26 09 INFO ha.ActiveStandbyElector 26 09:59:20 INFO ha.ActiveStandbyElector: Session connected.17/09/26 09:59:20 INFO ha.ActiveStandbyElector: Successfully created / hadoop-ha/ns in ZK.17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Session: 0x35ebc5163710000 closed17/09/26 09:59:20 INFO zookeeper.ClientCnxn: EventThread shut down

18. Formatted check

After the format is successful, you can see in the zookeeper

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.