Shulou(Shulou.com)06/01 Report--

This article mainly explains "the detailed steps of Hadoop cluster installation". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn the detailed steps of Hadoop cluster installation.

Docker installs Hadoop cluster

1. Stand-alone hadoop

First enter our existing container: (docker exec-it NAMES (name) COMMAND'/ bin/bash')

Mkdir / usr/hadoop

Cd / usr/hadoop

Decompress the uploaded hadoop, or wget to get the http://mirrors.sonic.net/apache/hadoop/common/hadoop-2.8.3/hadoop-2.8.3.tar.gz

Command: tar xvzf package (the editor generally deletes the package directly after decompression)

The preparatory work has been completed, and the construction of hadoop has begun.

Edit .bashrc

Vi / .bashrc

Add the following:

1.# Set Hadoop-related environment variables

Export HADOOP_HOME=/usr/local/hadoop

2.# Add Hadoop bin/ directory to PATH

Export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Refresh:

Source / .bashrc

There is a pit here:

If which cannot be found, execute: yum install which

Create three files:

Advanced cd / usr/local/hadoop/hadoop-2.7.2

Mkdir tmp as a temporary directory for Hadoop

Mkdir datanode as the storage directory of NameNode

Mkdir namenode as the storage directory of DataNode

Configuration file: cd / usr/local/hadoop/hadoop-2.7.2/etc/hadoop

Three files:

1.core-site.xml configuration:

Hadoop.tmp.dir

/ usr/local/hadoop/hadoop-2.7.2/tmp

Fs.default.name

Hdfs://master:9000

True

2.hdfs-site.xml configuration: (some configurations are prepared for clustering)

Dfs.replication

two

Dfs.namenode.name.dir

/ usr/local/hadoop/hadoop-2.7.2/namenode

True

Dfs.namenode.name.dir

/ usr/local/hadoop/hadoop-2.7.2/datanode

True

3.mapred-site.xml configuration: (there is a mapred-site.xml.template file, change to mapred-site.xml)

Mapred.job.tracker

Master:9001

Specify the JAVA_HOME environment variable as follows:

Using the command vi hadoop-env.sh, add the following configuration:

The java implementation to use. Export JAVA_HOME=/usr/java/jdk1.8.0_141

Then format the namenode

Execute the command: hadoop namenode-format

At this point, the hadoop configuration is complete.

Second, install SSH:

Use yum: yum install passwd openssl openssh-server-y

Because we are docker operation, it will not start automatically, so let's add it to it. In Bashrc

Execute the command to add the following: Vi ~ / .bashrc

# autorun

/ usr/sbin/sshd

Execute the command to generate the secret key and public key:

Ssh-keygen-t rsa-P''- f ~ / .ssh/id_dsa

Cd / .ssh/

Cat id_dsa.pub > > authorized_keys

Chmod 700. ssh/

Chmod 600 ~ / .ssh/authorized_keys

Supplement:

Unencrypted login problem: bash: ssh: command not found

Solution: yum-y install openssh-clients

At this location, we can save the container as a new image:

Docker commit-m "hadoop install" 61c9cf8da12f linux:hadoop

At this time, we have installed a stand-alone hadoop image.

Detect ssh:

Command 1./usr/sbin/sshd

Command 2.ssh

3. Docker hadoop cluster:

Remember: hostname and hosts configurations have been modified in the container and can only be valid for the current container lifecycle. If the container exits and restarts, the two configurations will be restored. And these two configurations cannot be written to the mirror through the commit command.

Configure hosts after startup of the three containers

Vi / etc/hosts Note to modify the ip address:

172.17.0.2 master172.17.0.3 slave1172.17.0.4 slave2 starts sshd / usr/sbin/sshd

Configure slaves:vi / usr/local/hadoop/hadoop-2.7.2/etc/hadoop/slaves

Will: master

Slave1

Slave2

After being added:

Start Hadoop: (enter / usr/local/hadoop/hadoop-2.7.2/sbin) start-all.sh to view-- "execute the jps command on the node."

Master is as follows:

Slave1 is as follows:

Slave2 is as follows:

You can also view the status of DataNode and NameNode through the Web page: http://IP:50070/:

At this point, I believe you have a deeper understanding of the "detailed steps of Hadoop cluster installation". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.