Shulou(Shulou.com)06/03 Report--

The alliance of HDFS (NameNode): Federation, configure HDFS alliance, multiple NameNode nodes survive at the same time, handle different requests of clients according to the routing rules configured by agents, achieve load balancing, and cache more meta-information at the same time. It is different from HA in that multiple NameNode are active, which can store more meta-information.

I. the connection and difference between Alliance and HA

1. There are also multiple NameNode

2. The difference from HA: these NameNode are active status.

3. Function:

(1) caching more meta-information

(2) realize load balancing

Second, build HDFS alliance

(1) Planning

NameNode:bigdata112 bigdata113

DataNode:bigdata114 bigdata115

(2) configure and install hadoop on bigdata112

(2.1) decompression

(2.2) configure the java environment in hadoop-env.sh

Export JAVA_HOME=/training/jdk1.8.0_144

(2.3) configure core-site.xml

Hadoop.tmp.dir / training/hadoop-2.7.3/tmp

(2.4) configure mapred-site.xml

Mapreduce.framework.name yarn (2.5) configure yarn-site.xml

Yarn.resourcemanager.hostname

Bigdata12

Yarn.nodemanager.aux-services

Mapreduce_shuffle

(2.6) configure slaves

Bigdata14 bigdata15

(2.7) configure hdfs-site.xml

Dfs.nameservices ns1 Ns2 dfs.namenode.rpc-address.ns1 bigdata112:9000 dfs.namenode.http-address.ns1 bigdata112:50070 Dfs.namenode.secondaryhttp-address.ns1 bigdata112:50090 dfs.namenode.rpc-address.ns2 bigdata113:9000 Dfs.namenode.http-address.ns2 bigdata113:50070 dfs.namenode.secondaryhttp-address.ns2 bigdata113:50090 dfs.replication 2 dfs.webhdfs.enabled true Dfs.permissions false

(2.8) configure core-site.xml and add routing rules

Fs.viewfs.mounttable.xdl1.homedir / home fs.viewfs.mounttable.xdl1.link./movies hdfs://bigdata114:9000/movies fs.viewfs.mounttable.xdl1.link./mp3 hdfs://bigdata115:9000/mp3 fs.default.name viewfs://xdl1

Note: if there are too many routing rules, you can create a separate xml file (mounTable.xml)

(3) copy the hadoop from 113,114,115 to 113,114,115

(4) format NameNode

Formatting on 112and 113th namenode nodes

Hdfs namenode-format-clusterId xdl1

(5) start

Start Hadoop on 112

Start-all.sh

(6) create a directory for the response on the DataNode node

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.