In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-16 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >
Share
Shulou(Shulou.com)06/01 Report--
In this issue, the editor will bring you about the process of building a fully distributed cluster in Apache Hadoop 2.8. the article is rich in content and analyzes and describes it from a professional point of view. I hope you can get something after reading this article.
Recently, I have set up an Apache Hadoop distributed cluster on my notebook computer, using the latest stable version 2.8. and equipped with NameNode and ResourceManager, HA is highly available, which is convenient for daily research and testing of Hadoop. The detailed construction process is as follows:
1. Install docker and create a docker container for building hadoop nodes
Docker is really a good thing. When you want to build a distributed cluster on your laptop, you can't virtualize too many nodes on the VMware because of the limited CPU, memory and disk. You can easily build a distributed cluster by using docker to create several containers.
(1) first install centos6.9 on VMware as a host, and then install docker. For details, please see another blog post: Centos6.9 installs docker.
(2) then pull the centos image from docker hub, which is used to create nodes of distributed clusters. It is recommended to install centos6 in docker (centos7 in docker has a pit, has been cheated, and whines). For more information, please see my other blog post: install centos6 in docker.
(3) after the centos image is ready, the docker container is created, which is used to build hadoop nodes.
# create 4 nodes to build hadoopdocker run-it-- name hadoopcentos1 centos:6 / bin/bashdocker run-it-- name hadoopcentos2 centos:6 / bin/bashdocker run-it-- name hadoopcentos3 centos:6 / bin/bashdocker run-it-- name hadoopcentos4 centos:6 / bin/bash# launch container docker start hadoopcentos1docker start hadoopcentos2docker start hadoopcentos3docker start hadoopcentos4
Note: at this time, turn off the firewall of the host and set the selinux to be unavailable.
# turn off the firewall chkconfig iptables offservice iptables stop# and set selinux to unavailable. Modify the SELINUX configuration item vi / etc/selinux/configSELINUX=disabled in the config file
2. Planning hadoop clusters
This time, four hadoop nodes are created, and NameNode HA and ResourceManager HA are implemented. The nodes are planned as follows.
Hadoop cluster node planning
Docker container ip address hostname node processes hadoopcentos1172.17.0.1hd1NameNode (active), JournalNode, Zookeeper, ZKFChadoopcentos2172.17.0.2hd2NameNode (standby), JournalNode, Zookeeper, ZKFC, NodeManager, DataNodehadoopcentos3172.17.0.3hd3ResourceManager (active), JournalNode, Zookeeper, NodeManager, DataNodehadoopcentos4172.17.0.4hd4ResourceManager (standby), NodeManager, DataNode
NameNode HA is deployed in hd1, hd2,ResourceManager HA is deployed in hd3 and hd4. Other processes are shown in the table above.
3. Configure the basic environment of Docker container
Since the centos 6 image pulled in Docker is a simplified version, and many instructions are not available, install some software first and configure the basic environment.
Enter a Docker container, such as hadoopcentos1
Docker exec-it hadoopcentos1 bin/bash
Replace the yum domestic source (per node)
Curl http://mirrors.aliyun.com/repo/Centos-6.repo > / etc/yum.repos.d/CentOS-Base-6-aliyun.repomv / etc/yum.repos.d/CentOS-Base.repo / etc/yum.repos.d/CentOS-Base.repo.bakyum clean allyum makecache
Install the relevant software (per node)
Yum install-y net-tools which openssh-clients openssh-server iproute.x86_64
Edit the sshd_config file and change the UsePAM to no
Vi / etc/ssh/sshd_config
Start ssh (per node)
Chkconfig sshd onservice sshd start
Create groups and accounts for installing apache hadoop
Groupadd ahadoopuseradd-m-g ahadoop ahadoop
Configure hostname mapping tabl
Vi / etc/hosts172.17.0.1 hd1.hdsite hd1172.17.0.2 hd2.hdsite hd2172.17.0.3 hd3.hdsite hd3172.17.0.4 hd4.hdsite hd4172.17.0.1 31d48048cb1e172.17.0.2 1620d6ed305d172.17.0.3 ed3702f8924e172.17.0.4 ee8319514df6
The random string in the last four lines is the hostname automatically generated by the docker container. Each docker container generated will automatically generate a hostname.
4. Install JDK
Download JDK 1.8 Linux x64 from oracle's website
Create the java directory and extract the installation (use root users so that other programs can also be used later)
Mkdir / usr/javacd / usr/javatar-zxvf jdk-8u131-linux-x64.tar.gz
5. Download apache hadoop
Download the latest stable version of apache hadoop 2.8from apache hadoop's website.
Switch to the ahadoop user, put hadoop 2.8in the / home/ahadoop directory, and extract it
Su ahadoopcd / home/ahadooptar-zxvf hadoop-2.8.0.tar.gz
6. Configure zookeeper
To achieve high availability of hadoop namenode HA and automatic fault switching, it is necessary to rely on zookeeper
Note: if you only achieve the high availability of namenode HA, you can not use zookeeper, as long as you configure the journalnode node of hadoop to achieve high availability. The main purpose of configuring zookeeper cluster is to monitor heartbeat and automatically switch in case of failure, which is an important goal for us to configure HA with high availability.
(1) download zookeeper
Go to apache zookeeper to officially download the latest version of zookeeper-3.4.10.tar.gz
(2) configure zookeeper
Before configuration, it is recommended to read the official configuration and installation introduction of apache zookeeper
A zookeeper cluster can only configure an odd number of nodes, such as 3, 5, 7... At least 3 or more are required, which is related to the guarantee mechanism of zookeeper. To ensure that most nodes are available, that is, (nmur1) / 2, the number of nodes must be odd.
Extract zookeeper (hd1,hd2,hd3)
Cd / home/ahadooptar-zxvf zookeeper-3.4.10.tar.gz
Create a zookeeper data file (hd1,hd2,hd3), in which different nodes are distinguished by myid, generally using 1, 2, 3...
Mkdir / home/ahadoop/zookeeper-dataecho'1' > / home/ahadoop/zookeeper-data/myidecho'2' > / home/ahadoop/zookeeper-data/myidecho'3' > / home/ahadoop/zookeeper-data/myid
Configure zoo.cfg Profil
Vi / home/ahadoop/zookeeper-3.4.10/conf/zoo.cfgdataDir=/home/ahadoop/zookeeper-data # modify zookeeper data directory clientPort=2181 # default port server.1=hd1:2888:3888server.2=hd2:2888:3888server.3=hd3:2888:3888
The last sequence number of server.x in server.1, server.2 and server.3 is the id in myid files of different nodes.
Now that zookeeper has been configured, start zookeeper (hd1,hd2,hd3) on each node using the zkServer.sh start command
You can check the status of a node using zkServer.sh status. The final Mode indicates the role of the node in the cluster. A zookeeper cluster has only one leader, and the rest is follower.
[ahadoop@31d48048cb1e ~] $zkServer.sh start & [1] 6855 [ahadoop@31d48048cb1e ~] $ZooKeeper JMX enabled by defaultUsing config: / home/ahadoop/zookeeper-3.4.10/bin/../conf/zoo.cfgStarting zookeeper. STARTED [1] + Done zkServer.sh start [ahadoop@1620d6ed305d ~] $[ahadoop@1620d6ed305d ~] $[ahadoop@1620d6ed305d ~] $zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: / home/ahadoop/zookeeper-3.4.10/bin/../conf/zoo.cfgMode: leader
7. Configure the configuration file of hadoop
Before configuring hadoop cluster, it is recommended to read the cluster configuration instructions on the official website of apache hadoop. Although it is an English document, it is easy to understand and help to further understand the knowledge of hadoop cluster. The official configuration is described below (p.s.) According to the configuration instructions in the official website, a small number of errors are found, which are not found until the actual configuration, which will be described below):
Official introduction to apache hadoop single Node configuration
Official introduction to apache hadoop Cluster configuration
Official introduction to apache hadoop namenode HA (QJM-based) configuration
Official introduction to apache hadoop resourcemanager HA configuration
(1) create the corresponding folder (each node)
Mkdir / home/ahadoop/hadoop-datamkdir / home/ahadoop/hadoop-data/namemkdir / home/ahadoop/hadoop-data/datamkdir / home/ahadoop/hadoop-data/checkpointmkdir / home/ahadoop/hadoop-data/tmpmkdir / home/ahadoop/hadoop-data/logmkdir / home/ahadoop/hadoop-data/journalnode
The home folder is hadoop-data, where:
Name: stores namenode data
Data: stores datanode data
Checkpoint: checkpoint data for namenode exists
Tmp: temporary fil
Log: storing logs
Journalnode: data that exists in jounalnode
(2) configure core-site.xml configuration file
The official provides a default core-default.xml configuration file for reference, but there are many contents that we can configure according to our needs.
Fs.defaultFS hdfs://hdcluster dfs.journalnode.edits.dir / home/ahadoop/hadoop-data/journalnode hadoop.tmp.dir / home/ahadoop/hadoop-data/tmp fs.trash.interval 1440 io.file.buffer.size 65536 ha.zookeeper.quorum hd1:2181,hd2:2181,hd3:2181
Among them
Fs.defaultFS: indicates the address of the cluster namenode. For namenode HA, you need to take a cluster id to distinguish it from other namenode of the hadoop federation. Here, hdcluster is taken as the ID of the cluster.
Dfs.journalnode.edits.dir, hadoop.tmp.dir: indicates the data file path and temporary file path of journalnode
Fs.trash.interval: indicates the retention time of the Recycle Bin (minutes), that is, the length of time retained in the Recycle Bin after hdfs files are deleted
Io.file.buffer.size: indicates the number of bytes read from the file (byte)
Ha.zookeeper.quorum: represents the host and port of the zookeeper cluster
(3) configure hdfs-site.xml configuration file
The official provides a default hdfs-default.xml configuration file for reference, which can be configured as follows
Dfs.replication 3 dfs.namenode.name.dir / home/ahadoop/hadoop-data/name dfs.blocksize 67108864 dfs.datanode.data.dir / home/ahadoop/hadoop-data/data dfs.namenode.checkpoint.dir / home/ahadoop/hadoop-data/checkpoint dfs.namenode.handler.count 10 dfs.datanode.handler.count 10 dfs.nameservices hdcluster dfs.ha.namenodes.hdcluster nn1 Nn2 dfs.namenode.rpc-address.hdcluster.nn1 hd1:9000 dfs.namenode.rpc-address.hdcluster.nn2 hd2:9000 dfs.namenode.http-address.hdcluster.nn1 hd1:50070 dfs.namenode.http-address.hdcluster.nn2 hd2:50070 dfs.namenode.shared.edits.dir qjournal://hd1:8485 Hd2:8485;hd3:8485/hdcluster dfs.client.failover.proxy.provider.hdcluster org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.automatic-failover.enabled.hdcluster true dfs.ha.fencing.methods shell (/ bin/true)
Where:
Dfs.replication: indicates the number of block backups for hdfs. Default is 3.
Dfs.namenode.name.dir,dfs.datanode.data.dir,dfs.namenode.checkpoint.dir: represents the data path of namenode, datanode, and checkpoint
Dfs.blocksize: indicates the size of the block. The default is 64m, which can be changed to 128m or even 256m as needed.
Dfs.namenode.handler.count, dfs.datanode.handler.count: indicates the number of processes in namenode and datanode
Dfs.nameservices: represents the ID of the cluster namenode, which is named hdcluster here. It should be consistent with the cluster ID in the configuration item fs.defaultFS in core-size.xml.
Dfs.ha.namenodes.hdcluster: represents the id of namenode, where there are two namenode nodes, so it is named using nn1,nn2
Dfs.namenode.rpc-address.hdcluster.nn1,dfs.namenode.rpc-address.hdcluster.nn2: represents the remote calling host and port of nn1 and nn2
Dfs.namenode.http-address.hdcluster.nn1,dfs.namenode.http-address.hdcluster.nn2: represents the node http service and port of nn1 and nn2
Dfs.namenode.shared.edits.dir: represents the metadata path shared by namenode. When configuring HA, use journalnode to save metadata and maintain the consistency of namenode metadata.
Dfs.client.failover.proxy.provider.hdcluster: a java class that indicates that a HDFS client connects to Active NameNode (default)
Dfs.ha.automatic-failover.enabled.hdcluster: indicates whether to switch automatically when the active namenode of namenode ha fails (set to true, of course, ^ ^)
Dfs.ha.fencing.methods: indicates the method used for automatic switching in the event of a failure
[knocking on the blackboard, attention students, the following is the key point] the official example configuration value is sshfence, but after experiments, using this method will not automatically switch, but when namenode active fails, namenode standby is still standby. Only when we log in to the namenode active fault node and restart the failed namenode, will the original namenode standby automatically switch to namenode active. ), this is not the purpose of configuring high availability at all.
According to the research, there are two methods of fencing: sshfence and shell. Where:
Sshfence method: refers to logging in to the active namenode node through ssh to kill the namenode process, so you also need to set ssh login without a password, and ensure the permission to kill the namenode process.
Shell method: means to run a shell script / command to prevent two namenode from being in active at the same time, and the script needs to be written by yourself. But in fact, QJM mode itself has fencing function, which ensures that only one namenode can write edits files to journalnode, so there is no need to set fencing method to achieve. However, when the failover occurs, the original active namenode may still be accepting read requests from the client, so the client is likely to read some outdated data (because the data for the new active namenode has been updated in real time). Therefore, it is recommended that you set the fencing method. If you really don't want to set the fencing method, you can set a method that returns success (without the fencing effect), such as "shell (/ bin/true)". This is purely for the successful return of the fencing method and does not need to have a real fencing effect. This improves the availability of the system and maintains its availability even if the fencing mechanism fails.
(4) configure mapred-site.xml
The default mapred-default.xml configuration file is officially provided for reference. Our configuration is as follows
Mapreduce.framework.name yarn
This configuration indicates the use of the yarn framework
(5) configure yarn-site.xml
The default yarn-default.xml configuration file is officially provided for reference, and combined with the official introduction of ResourceManager HA, the configuration is as follows
Yarn.nodemanager.aux-services mapreduce_shuffle yarn.resourcemanager.ha.enabled true yarn.resourcemanager.cluster-id hdcluster yarn.resourcemanager.ha.rm-ids rm1,rm2 yarn.resourcemanager.hostname.rm1 hd3 yarn.resourcemanager.hostname.rm2 hd4 yarn.resourcemanager.webapp.address.rm1 hd3:8088 yarn.resourcemanager.webapp.address.rm2 hd4:8088 yarn.resourcemanager.zk-address hd1:2181,hd2:2181,hd3:2181
Among them
The ancillary service running on yarn.nodemanager.aux-services:NodeManager must be configured as mapreduce_shuffle before running the MapReduce program, otherwise an error will be reported.
Yarn.resourcemanager.ha.enabled: indicates that the startup resourcemanager HA is highly available
Yarn.resourcemanager.cluster-id: represents the cluster ID of resourcemanager, which should not be confused with other clusters. It is named hdcluster here.
Yarn.resourcemanager.ha.rm-ids: represents the node id of resourcemanager, where there are two nodes, using rm1,rm2 as the ID
Yarn.resourcemanager.hostname.rm1,yarn.resourcemanager.hostname.rm2: indicates the host of rm1,rm2. Hd3 and hd4 are selected here.
Yarn.resourcemanager.webapp.address.rm1,yarn.resourcemanager.webapp.address.rm2: indicates the web page access address and port of rm1,rm2, that is, the job can be accessed through this address and port
Yarn.resourcemanager.zk-address: indicates the use of zookeeper to help manage zookeeper cluster hosts and ports of resourcemanager master and slave
(6) configure slave node hosts
Configure the slaves node host list of the hadoop distributed cluster, that is, the node on which the datanode is running. Take hd2, hd3, and hd4 here.
Vi / home/ahadoop/hadoop-2.8.0/etc/hadoop/slaves# configure slave host hd2hd3hd4
(7) configure log4j log log4j.properties
Modify the log save path of log4j and the granularity of log output as needed
The path to the vi / home/ahadoop/hadoop-2.8.0/etc/hadoop/log4j.properties# modification log hadoop.log.dir=/home/ahadoop/hadoop-data/log
(8) configure the environment variables of bash
Edit the .bash _ profile file and configure environment variables to facilitate daily execution of commands
Vi / home/ahadoop/.bash_profile# adds the following configuration export JAVA_HOME=/usr/java/jdk1.8.0_131export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=$PATH:$JAVA_HOME/binexport HADOOP_HOME=/home/ahadoop/hadoop-2.8.0export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbinexport ZOOKEEPER_HOME=/ at the end of .bash _ profile Home/ahadoop/zookeeper-3.4.10export PATH=$PATH:$ZOOKEEPER_HOME/bin
After configuration, use the source command to make it effective
Source / home/ahadoop/.bash_profile
8. Format hadoop namenode, and start hadoop distributed cluster
Through the above configuration file, the configuration of the hadoop cluster has been completed. When you run a hadoop cluster for the first time, you need to format the namenode before starting the entire cluster.
Note: you need to format namenode only when you run it for the first time. Otherwise, formatting will empty the metadata and data blocks of the entire cluster.
Convenient startup scripts are available in hadoop's sbin directory
The simplest scripts are start-all.sh and stop-all.sh, which can directly start / terminate the entire hadoop cluster. Using these two commands, you must implement ssh password-free login on cluster nodes. If you execute these commands on hd1 hosts, you must implement hd1 password-free login to hd2, hd3 and hd4. In a production environment, it is not recommended to use start-all.sh or stop-all.sh to maintain the entire cluster. What problems may occur in the startup / termination process that will affect the entire cluster. In the personal test section, go ahead, how to 6, how to come.
In addition, start-dfs.sh and stop-dfs.sh are provided to start / terminate hdfs,start-yarn.sh and stop-yarn.sh to start / terminate yarn. It is also necessary to implement ssh password-free login of cluster nodes, which is not recommended in production environments. In the personal test section, go ahead, how to 6, how to come.
Third, hadoop-daemon.sh,yarn-daemon.sh is also provided to start / terminate hdfs and yarn. The method is as follows: when you execute hadoop-daemon.sh start namenode, you start namenode (secondaryname, datanode, journalnode, etc.), and use stop to terminate; when you execute yarn-daemon.sh start resourcemanager, you start resourcemanager (similar to nodemanager), and you use stop to terminate. Using this startup mode, each node needs to start the corresponding process one by one, and there is no need to realize ssh password-free login. The advantage of this approach is that the process of a node starts / terminates, and if there is a problem, it will not affect the entire cluster, but the administrator must be very clear about the functional distribution of the nodes in the entire cluster.
Fourth, the new version of hadoop also provides the process startup / termination methods of hdfs and yarn (located in the bin directory of hadoop). The way to use hdfs namenode is to start namenode (secondaryname, datanode, journalnode, etc.). What if you want to terminate? Kill bar, kill corresponding process will automatically call the termination program; yarn resourcemanager, is to start resourcemanager (similar to nodemanager), if you want to terminate, the same kill is good. In this way, for beginners, it is possible to better understand the startup sequence of the entire cluster node.
In this test, hdfs and yarn are used to start the node
When using hadoop distributed cluster for the first time, you need to format namenode and synchronize the ha status to zookeeper. The startup sequence is as follows:
# format startup sequence for the first time # launch zookeeper (hd1,hd2,hd3) zkServer.sh start & # launch journalnode (hd1,hd2,hd3) hdfs journalnode & # format namenode (hd1) hdfs namenode-format# initialize HA state to zk (hd1) hdfs zkfc-formatZK & # launch namenode active (hd1) hdfs namenode & # synchronize namenode (hd2) hdfs namenode-bootstrapStandby# launch namenode standby (hd2) hdfs namenode & # launch ZookeeperFailoverController (hd1,hd2) hdfs zkfc & # launch datanode (hd2 Hd3,hd4) hdfs datanode & # launch resourcemanager (hd3,hd4) yarn resourcemanager & # launch nodemanager (hd2,hd3,hd4) yarn nodemanager &
After startup, use jps on each node to view the startup of the process
# hd1 [ahadoop@31d48048cb1e ~] $jps8976 NameNode8803 JournalNode9172 Jps9092 DFSZKFailoverController8750 QuorumPeerMain# hd2 [ahadoop@1620d6ed305d ~] $jps7428 QuorumPeerMain7636 NameNode8021 Jps7719 DFSZKFailoverController7784 DataNode7884 NodeManager7487 JournalNode# hd3 [ahadoop@ed3702f8924e ~] $jps4320 QuorumPeerMain4451 DataNode4900 Jps4772 NodeManager4373 JournalNode4540 ResourceManager# hd4 [ahadoop@ee8319514df6 ~] $jps4578 NodeManager4707 Jps4489 DataNode4508 ResourceManager
So far, the whole cluster has been started successfully.
If you don't have to format namenode when you use it later (otherwise the data will be destroyed), then normally, the order in which the cluster starts is
# Boot sequence for daily use # launch zookeeper (hd1,hd2,hd3) zkServer.sh start & # launch journalnode (hd1,hd2,hd3) hdfs journalnode & # launch namenode active (hd1) hdfs namenode & # launch namenode standby (hd2) hdfs namenode & # launch ZookeeperFailoverController (hd1,hd2) hdfs zkfc & # launch datanode (hd2,hd3,hd4) hdfs datanode & # launch resourcemanager (hd3,hd4) yarn resourcemanager & # launch nodemanager (hd2,hd3,hd4) yarn nodemanager &
9. Hadoop cluster test
After the cluster starts, enter the following URL in the browser to check the status of namenode and resourcemanager
Enter the URL http://172.17.0.1:50070 to view namenode (active) node information
Enter the URL http://172.17.0.2:50070 to view namenode (standby) node information
Enter the URL http://172.17.0.3:8088 to view resourcemanager node information
Use the example of the official website to test the map-reduce of the cluster
This example uses the map-reduce jar package that comes with hadoop to count the configuration items in etc/hadoop 's xml configuration file (similar to the classic wordcount test)
The specific steps are as follows:
# create the hdfs directory hdfs dfs-mkdir / userhdfs dfs-mkdir / user/ahadoophdfs dfs-mkdir input# to upload the xml configuration file hdfs dfs-put / home/ahadoop/hadoop-2.8.0/etc/hadoop/*.xml input# of etc/hadoop and execute the self-contained mapreduce jar program hadoop jar / home/ahadoop/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar grep input output 'dfs [a Murz.] + '# get the execution result hdfs dfs-get output output# to view the execution result cat output/*
After execution, the results are as follows
As can be seen from the figure above, map reduce is used to count the number of configuration items.
10. HA test of hadoop cluster
The HA high availability of the hadoop cluster is an important goal of this cluster. The namenode ha and resourcemanager ha of the hadoop cluster will be tested below.
(1) namenode ha test
At present, the namenode of hd1 is active, and the namenode of hd2 is standby. Simply and rudely kill the namenode of hd1, that is, make the namenode of hd1 malfunction. At this time, if you look at the namenode of hd2, you will find that it has been automatically switched to active.
It indicates that the automatic switchover of namenode ha is successful.
(2) resourcemanager ha test
The resourcemanager ha of hd3 and hd4 does not see active or standby status after opening the web page. If hd3 is active status, enter http://172.17.0.3:8088 to see the resourcemanger page. When you enter http://172.17.0.4:8088, it automatically switches back to the htttp://172.17.0.3:8088 page.
Here, you can view the status of the two resourcemanager through the background. The command is as follows
# View resourcemanager status [ahadoop@ed3702f8924e ~] $yarn rmadmin-getServiceState rm1 active [ahadoop@ed3702f8924e ~] $yarn rmadmin-getServiceState rm2 standby
When testing the resourcemanager ha, kill the resourcemanager of the active node, and then check the rm2 status, and you will find that the active status has been variable.
[ahadoop@ed3702f8924e ~] $yarn rmadmin-getServiceState rm2 active
You can use the following instructions to switch the status of active and standby (--forcemanual means mandatory)
# switch resourcemanager master / slave # switch to standby status yarn rmadmin-transitionToStandby-- forcemanual rm2# switch to active status yarn rmadmin-transitionToActive-- forcemanual rm1
Note that if the active state is forced to switch to standby, it can be switched successfully, that is, both nodes are in standby state. However, if there is already an active state, and then another standby state is switched to active state, the switch cannot be carried out, and the system prompts that there is already a node in active state.
[ahadoop@ee8319514df6] $yarn rmadmin-transitionToActive-- forcemanual rm2You have specified the-- forcemanual flag. This flag is dangerous, as it can induce a split-brain scenario that WILL CORRUPT your HDFS namespace, possibly irrecoverably.It is recommended not to use this flag, but instead to shut down the cluster and disable automatic failover if you prefer to manually manage your HA state.You may abort safely by answering 'n'or hitting ^ C now.Are you sure you want to continue? (Y or N) Y17 WARN ha.HAAdmin 06 WARN ha.HAAdmin: Proceeding with manual HA state management even thoughautomatic failover is enabled for org.apache.hadoop.yarn.client.RMHAServiceTarget@3444d69d17/06/18 16:53:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... 18 16:53:01 Using builtin-java classes where applicabletransitionToActive: Node rm1 is already activeUsage: rmadmin [- transitionToActive [--forceactive]]
11. Hadoop distributed cluster is easy to build.
The above configuration is for namenode ha and resourcemanager ha, and it will be complicated. If you just want to build a hadoop distributed cluster without configuring namenode ha or resourcemanager ha for the time being, the configuration will be much simpler. The following shows the configuration of hadoop distributed cluster.
(1) core-site.xml
Fs.defaultFS hdfs://hd1:9000 hadoop.tmp.dir / home/ahadoop/hadoop-data/tmp fs.trash.interval 1440 io.file.buffer.size 65536
(2) hdfs-site.xml
Dfs.replication 3 dfs.namenode.name.dir / home/ahadoop/hadoop-data/name dfs.blocksize 67108864 dfs.datanode.data.dir / home/ahadoop/hadoop-data/data dfs.namenode.checkpoint.dir / home/ahadoop/hadoop-data/checkpoint dfs.namenode.handler.count 10 dfs.datanode.handler.count 10
(3) mapred-site.xml
Mapreduce.framework.name yarn
(4) yarn-site.xml
Yarn.resourcemanager.hostname hd3 yarn.nodemanager.aux-services mapreduce_shuffle
(5) slaves
Hd2hd3hd4
(6). Bash_profile
# set the environment variable vi / home/ahadoop/.bash_profileexport JAVA_HOME=/usr/java/jdk1.8.0_131export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport PATH=$PATH:$JAVA_HOME/binexport HADOOP_HOME=/home/ahadoop/hadoop-2.8.0export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
(7) format namenode and start hadoop cluster
# format namenode (hd1) hdfs namenode-format# launch namenode (hd1) hdfs namenode & # launch secondarynamenode (hd2) hdfs secondarynamenode & # launch datanode (hd2, hd3, hd4) hdfs datanode & # launch resourcemanager (hd2) yarn resoucemanager & # launch nodemanager (hd2, hd3, hd4) hdfs nodemanager &
(8) visit namenode and resourcemanager pages
Visit the namenode page
Http://172.17.0.1:50070
Visit the resourcemanager page
Http://172.17.0.2:8088
12. Some problems in the official apache hadoop configuration document
(1) the namenode ha configuration document given by the official website, in which the configuration of automatic handover failure is (hdfs-site.xml).
Dfs.ha.automatic-failover.enabled true
In fact, the cluster ID is added after the configuration. The cluster ID in this test is hdcluster, so the normal configuration is
Dfs.ha.automatic-failover.enabled.hdcluster true
(2) the question of starting the command
The hadoop cluster configuration documentation given on the official website, in which the startup command for the node process is:
[hdfs] $HADOOP_HOME/bin/hdfs-- daemon start namenode [hdfs] $$HADOOP_HOME/bin/hdfs-- daemon start datanode [yarn] $$HADOOP_HOME/bin/yarn-daemon start resourcemanager [yarn] $$HADOOP_HOME/bin/yarn-- daemon start nodemanager
However, it cannot be executed (or it may be a 3.0 beta version of the instruction, which has not been tried), and execution will prompt
[ahadoop@31d48048cb1e] $hdfs-- daemon start namenodeUnrecognized option:-- daemonError: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.
The parameter-- daemon is not supported, so when actually starting these node processes, the startup command is
[hdfs] $HADOOP_HOME/bin/hdfs namenode [hdfs] $HADOOP_HOME/bin/hdfs datanode [yarn] $$HADOOP_HOME/bin/yarn resourcemanager [yarn] $$HADOOP_HOME/bin/yarn nodemanager above is what the Apache Hadoop 2.8 fully distributed cluster building process is like. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.