Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

Building hadoop Cluster (2) High availability of YARN

2025-04-05 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Share

Shulou(Shulou.com)06/03 Report--

[root@hadoop01 ~]# vim /usr/local/hadoop-2.7.1/etc/hadoop/yarn-site.xml

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.resourcemanager.ha.enabled

true

yarn.resourcemanager.ha.automatic-failover.enabled

true

yarn.resourcemanager.cluster-id

qianfengyarn

yarn.resourcemanager.ha.rm-ids

rm1,rm2

yarn.resourcemanager.hostname.rm1

hadoop01

yarn.resourcemanager.hostname.rm2

hadoop02

yarn.resourcemanager.webapp.address.rm1

hadoop01:8088

yarn.resourcemanager.webapp.address.rm2

hadoop02:8088

yarn.resourcemanager.zk-address

hadoop01:2181,hadoop02:2181,hadoop03:2181

[root@hadoop01 hadoop-2.7.1]# vim ./ etc/hadoop/mapred-site.xml

mapreduce.framework.name

yarn

true

mapreduce.map.memory.mb

1024

mapreduce.jobhistory.address

hadoop01:10020

mapreduce.jobhistory.webapp.address

hadoop01:19888

Send to cluster server

[root@hadoop01 hadoop-2.7.1]# scp -r ./ etc/hadoop/mapred-site.xml hadoop02:/usr/local/hadoop-2.7.1/etc/hadoop/

mapred-site.xml 100% 1218 1.2KB/s 00:00

[root@hadoop01 hadoop-2.7.1]# scp -r ./ etc/hadoop/mapred-site.xml hadoop03:/usr/local/hadoop-2.7.1/etc/hadoop/

mapred-site.xml 100% 1218 1.2KB/s 00:00

[root@hadoop01 hadoop-2.7.1]# scp -r ./ etc/hadoop/yarn-site.xml hadoop03:/usr/local/hadoop-2.7.1/etc/hadoop/

yarn-site.xml 100% 1845 1.8KB/s 00:00

[root@hadoop01 hadoop-2.7.1]# scp -r ./ etc/hadoop/yarn-site.xml hadoop02:/usr/local/hadoop-2.7.1/etc/hadoop/

yarn-site.xml

[root@hadoop01 hadoop-2.7.1]# start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-resourcemanager-hadoop01.out

hadoop01: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop01.out

hadoop03: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop03.out

hadoop02: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop02.out

[root@hadoop02 sbin]# yarn-daemon.sh start resourcemanager

starting resourcemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-resourcemanager-hadoop02.out

View ResourceManager status

[root@hadoop01 hadoop-2.7.1]# yarn rmadmin -getServiceState rm1

18/01/13 16:17:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

active

[root@hadoop02 sbin]# yarn rmadmin -getServiceState rm2

18/01/13 16:20:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

standby

[root@hadoop01 ~]# start-all.sh

[root@hadoop01 ~]# service ntpd start

Cluster all machines start zookeeper, zkServer.sh start

[root@hadoop01 ~]# yarn-daemon.sh start resourcemanager

[root@hadoop01 ~]# start-dfs.sh

All processes must be present, please check carefully

[root@hadoop01 logs]# jps

8608 JobHistoryServer

16833 DataNode

3155 QuorumPeerMain

17300 ResourceManager

17463 Jps

16727 NameNode

17210 DFSZKFailoverController

17423 NodeManager

17055 JournalNode

[root@hadoop02 ~]# jps

9472 DFSZKFailoverController

7108 ResourceManager

9254 DataNode

9719 Jps

9353 JournalNode

9566 NodeManager

9183 NameNode

2751 QuorumPeerMain

[root@hadoop03 ~]# jps

2546 QuorumPeerMain

6738 Jps

6503 JournalNode

6408 DataNode

6606 NodeManager

8608 JobHistoryServer

[root@hadoop01 logs]# mr-jobhistory-daemon.sh stop historyserver

[root@hadoop01 logs]# mr-jobhistory-daemon.sh start historyserver

7108 ResourceManager

[root@hadoop02 ~]# stop-yarn.sh

stopping yarn daemons

stopping resourcemanager

hadoop02: no nodemanager to stop

hadoop01: no nodemanager to stop

hadoop03: no nodemanager to stop

no proxyserver to stop

[root@hadoop02 sbin]# yarn-daemon.sh start resourcemanager

[root@hadoop02 ~]# hadoop-daemon.sh start namenode

starting namenode, logging to /usr/local/hadoop-2.7.1/logs/hadoop-root-namenode-hadoop02.out

test

hadoop jar /usr/local/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /README.txt /out/05

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 0

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Internet Technology

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report