Shulou(Shulou.com)06/03 Report--

Today, the editor will bring you an article on configuring Hadoop HA cluster files. The editor thinks it's pretty good, so I'll share it for you as a reference. Let's follow the editor and have a look.

> 1. Hadoop-env.sh configuration

Export JAVA_HOME= (jdk installation address)

> 2. Hdfs-site.xml configuration file

Dfs.nameservices

Mycluster

Dfs.ha.namenodes.mycluster

Master,slave1

Dfs.namenode.rpc-address.mycluster.master

Master:50070

Dfs.namenode.rpc-address.mycluster.slave1

Slave1:50070

Dfs.namenode.http-address.mycluster.master

Master:9000

Dfs.namenode.http-address.mycluster.slave1

Slave1:9000

Dfs.namenode.shared.edits.dir

Qjournal://master:8485;slave1:8485;slave2:8485/mycluster

Dfs.client.failover.proxy.provider.mycluster

Org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

Dfs.ha.fencing.methods

Sshfence

Dfs.ha.fencing.ssh.private-key-files

/ root/.ssh/id_rsa

Dfs.ha.fencing.methods

Shell (/ bin/true)

Dfs.journalnode.edits.dir

/ usr/local/src/hadoop/data/journalnode/jn

Dfs.ha.automatic-failover.enabled

True

> 3. Core-site.xml configuration file

Fs.defaultFS

Hdfs://mycluster

Hadoop.tmp.dir

/ opt/tmp/hadoop/ha

Ha.zookeeper.quorum

Master:2181,slave1:2181,slave2:2181

> 4. Mapred-site.xml configuration

Mapreduce.farmework.name

Yarn

> 5. Yarn-site.xml configuration

Yarn.resourcemanager.ha.enabled

True

Yarn.resourcemanager.ha.automatic-failover.enabled

True

Yarn.resourcemanager.cluster-id

Yarn-rm-cluster

Yarn.resourcemanager.ha.rm-ids

Rm1,rm2

Yarn.resourcemanager.hostname.rm1

Master

Yarn.resourcemanager.hostname.rm2

Server1

Yarn.resourcemanager.zk-address

Master:2181,server1:2181,server2:2181

Yarn.resourcemanager.webapp.address.rm1

Master:8088

Yarn.resourcemanager.webapp.address.rm2

Slave1:8088

Yarn.nodemanager.aux-services

Mapreduce_shuffle

> 6. Slaves configuration (DataNode)

Master

Slave1

Slave2

The above is a summary of how to configure Hadoop HA cluster files, which is more comprehensive, and I also believe that there are quite a few tools that we may see or use in our daily work. Through this article, I hope you can gain more.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.