Shulou(Shulou.com)06/03 Report--

0. Preface

The purpose of this paper is to use a newly installed Linux system to build a Hadoop pseudo-distributed environment from 0 to achieve the purpose of building quickly, so as to experience the charm of Hadoop and provide a basic environment for further learning.

The system environment used is described as follows:

Operating system: CentOS 6.564 bit

Host IP address: 10.0.0.131max 24

Hostname: leaf

User name: root

Hadoop version: 2.6.5

Jdk version: 1.7

As you can see, instead of following most tutorials to create a hadoop user to operate, the purpose of using root users directly is to quickly build a Hadoop environment for experience.

In order to ensure that the later operations can be completed normally, please confirm whether the host name leaf can be resolved on this machine. If not, manually add the resolution to the / etc/hosts directory:

[root@leaf ~] # echo "127.0.0.1 leaf" > > / etc/hosts [root@leaf ~] # ping leafPING leaf (127.0.0.1) 56 (84) bytes of data.64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.043 ms64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.048 ms64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.046 ms

1.rsync software installation

Use the following command to install:

[root@leaf ~] # yum install-y rsync

2.ssh installation and password-free login configuration

(1) ssh installation

Install using the following command

[root@leaf ~] # yum install-y openssh-server openssh-clients

(2) password-free login configuration for ssh

Because Hadoop uses the ssh protocol to manage remote daemons, you need to configure password-free login.

Turn off the firewall and selinux

To ensure successful configuration, turn off the firewall and selinux before configuring:

# turn off firewall [root@leaf ~] # / etc/init.d/iptables stop [root@leaf ~] # chkconfig-- level 3 iptables off# turn off selinux [root@leaf ~] # setenforce 0 [root@leaf ~] # sed-is / SELINUX=enforcing/SELINUX=disabled/g / etc/selinux/config [root@leaf ~] # cat / etc/selinux/config | grep disabled# disabled-No SELinux policy is loaded.SELINUX=disabled

Generate key pair

[root@leaf ~] # mkdir .ssh [root@leaf ~] # ssh-keygen-t dsa-P''- f .ssh / id_dsaGenerating public/private dsa key pair.Your identification has been saved in .ssh / id_dsa.Your public key has been saved in .ssh / id_dsa.pub.The key fingerprint is:5b:af:7c:45:f3:ff:dc:50:f5:81:4b:1e:5c:c1:86:90 root@leafThe key's randomart p_w_picpath Is:+-- [DSA 1024]-+ | .o oo. | | E..oo | =. | o = + | | S. + oo | | o. . | |. . | |. .. Oo | | o. = | +-+

Add the public key to the local trust list

[root@leaf ~] # cat .ssh / id_dsa.pub > > .ssh / authorized_keys

Verification

After the above three steps are completed, the configuration of password-free login is completed, and you can verify it with the following command:

[root@leaf] # ssh localhostThe authenticity of host 'localhost (:: 1)' can't be established.RSA key fingerprint is d1:0d:ed:eb:e7:d1:2f:02:23:70:ef:11:14:4e:fa:42.Are you sure you want to continue connecting (yes/no)? YesWarning: Permanently added 'localhost' (RSA) to the list of known hosts.Last login: Wed Aug 30 04:28:01 2017 from 10.0.0.1 [root@leaf] #

You need to enter yes when you log in for the first time, and then you can log in directly when you log in:

[root@leaf] # ssh localhostLast login: Wed Aug 30 04:44:02 2017 from localhost [root@leaf] #

3.jdk installation and configuration

(1) download jdk

The jdk1.7 version is used here, which can be downloaded from the following website:

Http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html

After the download is completed, you can upload it to the / root directory using winscp, as follows:

[root@leaf] # ls-lh jdk-7u80-linux-x64.tar.gz-rw-r--r--. 1 root root 147M Aug 29 12:05 jdk-7u80-linux-x64.tar.gz

(2) jdk installation

Extract the jdk to the / usr/local directory and create a soft link:

[root@leaf ~] # cp jdk-7u80-linux-x64.tar.gz / usr/local/ [root@leaf ~] # cd / usr/local/ [root@leaf local] # tar-zxf jdk-7u80-linux-x64.tar.gz [root@leaf local] # ls-ld jdk1.7.0_80/drwxr-xr-x. 8 uucp 4096 Apr 11 2015 jdk1.7.0_80/ [root @ leaf local] # ln-s jdk1.7.0_80/ jdk [root@leaf local] # ls-ld jdklrwxrwxrwx. 1 root root 12 Aug 30 04:56 jdk-> jdk1.7.0_80/

(3) JAVA_HOME environment variable configuration

The java command is in the / usr/local/jdk/bin directory:

[root@leaf local] # cd jdk/bin/ [root@leaf bin] # ls-lh java-rwxr-xr-x. 1 uucp 143 7.6K Apr 11 2015 java

Configure the java environment variable:

[root@leaf bin] # echo 'export JAVA_HOME=/usr/local/jdk/bin' > > / etc/profile [root@leaf bin] # echo' export PATH=$PATH:$JAVA_HOME' > > / etc/profile [root@leaf bin] # source / etc/profile

This allows us to use java-related commands in any directory:

[root@leaf ~] # java-versionjava version "1.7.0,80" Java (TM) SE Runtime Environment (build 1.7.0_80-b15) Java HotSpot (TM) 64-Bit Server VM (build 24.80-b11, mixed mode) [root@leaf ~] # javac-versionjavac 1.7.080

4.hadoop installation and configuration

(1) download hadoop

Version 2.6.5 of hadoop is used here, which can be downloaded from the following website:

Http://hadoop.apache.org/releases.html

Select 2.6.5 binary to enter the appropriate page to download, and then upload it to the / root directory using winscp, as follows:

[root@leaf] # ls-lh hadoop-2.6.5.tar.gz-rw-r--r--. 1 root root 191M Aug 29 19:09 hadoop-2.6.5.tar.gz

(2) hadoop installation

Extract the hadoop to the / usr/local directory and create a soft link:

[root@leaf ~] # cp hadoop-2.6.5.tar.gz / usr/local [root@leaf ~] # cd / usr/local [root@leaf local] # tar-zxf hadoop-2.6.5.tar.gz [root@leaf local] # ls-ld hadoop-2.6.5drwxrwxr-x. 9 1000 1000 4096 Oct 3 2016 hadoop-2.6.5 [root@leaf local] # ln-s hadoop-2.6.5 hadoop [root@leaf local] # ls-ld hadooplrwxrwxrwx. 1 root root 12 Aug 30 05:05 hadoop- > hadoop-2.6.5

(3) hadoop environment variable configuration

The hadoop-related commands are in the / usr/local/hadoop/bin and / usr/local/hadoop/sbin directories, as follows:

[root@leaf local] # cd hadoop/bin/ [root@leaf bin] # ls-lh hadoop-rwxr-xr-x. 1 1000 1000 5.4K Oct 3 2016 hadoop

Configure the hadoop environment variable:

[root@leaf bin] # echo 'export HADOOP_HOME=/usr/local/hadoop/bin:/usr/local/hadoop/sbin' > > / etc/profile [root@leaf bin] # echo' export PATH=$PATH:$HADOOP_HOME' > > / etc/profile

This allows us to use hadoop-related commands in any directory:

[root@leaf ~] # hadoopUsage: hadoop [--config confdir] COMMAND where COMMAND is one of: fs run a generic filesystem user client version print the version jar run a jar file checknative [- a |-h] checknative hadoop and compression libraries availability distcp copy file or directories recursively archive-archiveName NAME-p * create a hadoop archive classpath prints the classpath needed to get the credential interact With credential providers Hadoop jar and the required libraries daemonlog get/set the log level for each daemon trace view and modify Hadoop tracing settings or CLASSNAME run the class named CLASSNAMEMost commands print help when invoked w/o parameters.

(4) hadoop configuration

The configuration file for hadoop is in the / usr/local/hadoop/etc/hadoop directory:

[root@leaf] # cd / usr/local/hadoop/etc/hadoop/ [root@leaf hadoop] # lscapacity-scheduler.xml hadoop-policy.xml kms-log4j.properties ssl-client.xml.exampleconfiguration.xsl hdfs-site.xml kms-site.xml ssl-server.xml.examplecontainer-executor.cfg httpfs-env.sh log4j.properties yarn-env. Cmdcore-site.xml httpfs-log4j.properties mapred-env.cmd yarn-env.shhadoop-env.cmd httpfs-signature.secret mapred-env.sh yarn-site.xmlhadoop-env.sh httpfs-site.xml mapred-queues.xml.templatehadoop-metrics2.properties kms-acls.xml mapred-site.xml.templatehadoop-metrics.properties kms-env.sh slaves

Configure core-site.xml

Fs.default.name hdfs://localhost:9000

The value under the fs.default.name field is used to specify the IP address and port number of NameNode (Master of HDFS). For example, the value value hdfs://localhost:9000 below indicates that the IP address or host of HDFS NameNode is localhost, and the port number is 9000.

Configure hdfs-site.xml

Dfs.replication 1 dfs.name.dir / root/hdfs-filesystem/name dfs.data.dir / root/hdfs-filesystem/data

Dfs.replication is used to specify the number of times each Block block in HDFS is copied, which plays the role of data redundancy backup; dfs.name.dir is used to configure the metadata of NameNode of HDFS, separated by commas, and HDFS will copy metadata redundancy to these directories; dfs.data.dir is used to configure the data directory of HDFS's DataNode, separated by commas, and HDFS will store the data in these directories.

Configure mapred-site.xml

Mapred.job.tracker localhost:9001

The mapred.job.tracker field is used to specify the IP address and port number of the MapReduce Jobtracker, for example, the IP address or the host is localhost,9001 is the interactive port of MapReduce Jobtracker RPC.

Configure hadoop-env.sh

Export JAVA_HOME=/usr/local/jdk

5.hadoop startup and testing

(1) format HDFS distributed file system

Execute the following command:

[root@leaf ~] # hadoop namenode-format...17/08/30 08:41:29 INFO namenode.NNStorageRetentionManager: Going to retain 1 p_w_picpaths with txid > = 017Exiting with status 08 hadoop namenode 30 08:41:29 INFO util.ExitUtil: Exiting with status 017 Greater 08 08:41:29 INFO namenode.NameNode: SHUTDOWN_MSG: / * * * SHUTDOWN_MSG: Shutting down NameNode at leaf/127.0.0.1***/

Note that the output display is similar to the one above, and if so, the operation is successful.

(2) start the hadoop service

Execute the following command:

[root@leaf] # start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh17/08/30 08:53:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... Using builtin-java classes where applicableStarting namenodes on [localhost] localhost: starting namenode, logging to / usr/local/hadoop-2.6.5/logs/hadoop-root-namenode-leaf.outlocalhost: starting datanode, logging to / usr/local/hadoop-2.6.5/logs/hadoop-root-datanode-leaf.outStarting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode Logging to / usr/local/hadoop-2.6.5/logs/hadoop-root-secondarynamenode-leaf.out17/08/30 08:53:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... Using builtin-java classes where applicablestarting yarn daemonsstarting resourcemanager, logging to / usr/local/hadoop-2.6.5/logs/yarn-root-resourcemanager-leaf.outlocalhost: starting nodemanager, logging to / usr/local/hadoop-2.6.5/logs/yarn-root-nodemanager-leaf.out

(3) hadoop service testing

After the startup is complete, execute the jps command, and you can see the daemon that hadoop is running, as follows:

[root@leaf ~] # jps4167 SecondaryNameNode4708 Jps3907 NameNode4394 NodeManager4306 ResourceManager3993 DataNode

You can also access the relevant page by entering the address in the browser. Here, you can visit the page of NameNode, whose address is http://10.0.0.131:50070, as follows:

Visit the DataNode page at http://10.0.0.131:50075, as follows

6. references

"Hadoop Core Technology"

It is important to note, however, that the version of the book is 1.x, and here is version 2.x.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.