Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to build a Hadoop cluster". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to build a Hadoop cluster.

First, the basic concept of Hadoop cluster.

There are three operation modes of Hadoop: local operation mode, pseudo-distributed operation mode and cluster operation mode. The corresponding concepts are as follows:

1. Stand-alone mode, that is, local running mode (standalone or local mode) does not need to run any daemon (daemon), all programs are executed on a single JVM. Because it is convenient to test and debug MapReduce programs in native mode, this mode is suitable for the development phase. Stand-alone mode is the default mode for Hadoop. When the source package of Hadoop was extracted for the first time, Hadoop could not understand the hardware installation environment, so it conservatively chose the minimum configuration. All three XML files are empty in this default mode. When the configuration file is empty, Hadoop runs entirely locally. Because there is no need to interact with other nodes, stand-alone mode does not use HDFS and does not load any Hadoop daemons. This mode is mainly used to develop and debug the application logic of MapReduce programs.

2. Pseudo-distributed running mode: if the Java processes corresponding to Hadoop are running on a physical machine, it is called pseudo-distributed running mode.

3. Cluster mode. If the Java process corresponding to Hadoop is running on multiple physical machines, it is called cluster mode.

Second, the steps of building Hadoop cluster.

1. Server preparation

In this case, virtual machine servers are used to build HADOOP clusters, and the software and version used are as follows:

Vmware 14.0

Centos 6.7 64bit

two。 Configure the network card

Terminal input vi / etc/sysconfig/neywork-scripts/ifcfg-eth0

Enter IP address, subnet mask, network card, domain name server

3. Test whether the network card is configured correctly

Restart the network service enter service network restart

Enter ping www.baidu.com if you can ping, the Nic will be configured successfully.

4. Modify hostname

Enter vi / etc/sysconfig/network to modify the hostname

5. Turn off the firewall

Service iptables stop only shut down once.

Chkconfig iptables off shuts down permanently

6. Install the ssh client

Execute yum install-y openssh-clients

7. Clone server

Close the virtual machine, right-click > manage > Clone, and select full Clone

Configure the Nic for the cloned machine and execute vi / etc/udev/rules.d/70-persistent-net.rules

Execute vi / etc/sysconfig/neywork-scripts/ifcfg-eth0 and modify the IP address.

Execute vi / etc/sysconfig/network and change the hostname to hadoop02

Repeat the above steps to set up a hadoop03 server

Restart each machine for the network card to take effect

8.hosts mapping

Execute vi / etc/hosts, enter the following, and enter the following under C:\ Windows\ System32\ drivers\ etc\ hosts of Windows

9.ssh link and secret-free login

Type ssh hadoop02 on the hadoop01 server to test whether the ssh connection is successful

Create a new vi Auto.sh file on the hadoop01 server and write a secret-free script in the file

Execute the script to achieve secret-free login (only a single secret-free login)

10. Install JDK

1: upload the JDK installation package to linux

2: extract the files to the installation directory tar-zxvf / root/jdk-8u102-linux-x64.tar.gz-C / usr/local/

3: configure the environment variable vi / etc/profile, enter at the end

Export PATH=$PATH:$JAVA_HOME/bin

11. Install Hadoop

1: upload hadoop installation package

2: extract the files to the installation directory tar-zxvf / root/hadoop-2.8.3.tar.gz-C / usr/local/

3: enter the hadoop installation directory and start the configuration file under cd hadoop-2.8.3/etc/hadoop

4: where is the execution vi / etc/hadoop/hadoop-env.sh JAVA_HOME

The java implementation to use.

Export JAVA_HOME=/usr/local/jdk1.7.0_51

Change export JAVAHOME to absolute path

5: execute vi core-site.xml

6: vi hdfs-site.xml

7: perform renaming cp mapred-site.xml.tmp* mapred-site.xml

Vi mapred-site.xml

8: vi yarn-site.xml

9: vi salves

10: configure the environment variable vi / etc/profile

twelve。 JDK and hadoop installation of the other two servers

Send the first installed jdk and hadoop and the configuration file to the other two

Hosts file scp-r / etc/hosts/ hadoop02:/etc/

Folder scp-r / usr/local/jdk1.8.0_102 hadoop02:/usr/local/ after jdk installation

Folder scp-r / usr/local/hadoop-2.8.3 hadoop02:/usr/local/ after Hadoop installation

/ etc/profile configuration file scp-r / etc/profile hadoop02:/etc/

Hadoop03 does the same thing.

13. Start the Hadoop cluster

Initialize HDFS (operate on hadoop01)

Bin/hadoop namenode-format

Start HDFS

Start YARN

The hadoop cluster has been built successfully!

Third, matters needing attention in building Hadoop clusters.

There will inevitably be some problems in the process of building hadoop for the first time. For example, the cluster suddenly shows that there is a problem with the virtual machine, resulting in the loss of one DataNode data and the inability to remotely access other DataNode hosts and virtual machines through Xshell. If all the solutions that can be found on the Internet can be found, but still can not be solved, the virtual machine can be deleted and a machine can be cloned again.

At this point, I believe you have a deeper understanding of "how to build a Hadoop cluster". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.