Shulou(Shulou.com)06/03 Report--

-- Software version--

RHEL6.8hadoop2.8.1apache-maven-3.3.9findbugs-1.3.9protobuf-2.5.0.tar.gzjdk-8u45

-- Software version--

1.Hadoop

Macroscopic: Hadoop-based biosphere hadoop flume.

Narrow sense: apache hadoop hadoop.apache.org

2.Hadoop (Storage + Computing + Resource and Job scheduling)

Hadoop1.x

HDFS storage

MapReduce Computing + Resource and Job scheduling

Hadoop2.x enterprise is using

HDFS storage

MapReduce calculation

Both YARN resources and job scheduling platform computing components will on yarn

Hadoop3.x?

EC technology: Erasure Encoding, referred to as EC, is a new feature extended by Hadoop3 to HDFS, which is used to solve storage space files.

YARN: provides YARN's timeline service V.2 so that users and developers can test it and provide feedback.

Optimize Hadoop Shell script

Refactoring Hadoop Client Jar packages

Support for random Container

MapReduce task-level local optimization

Support for multiple NameNode

Some default service ports have been changed

Support for file system connectors

Load balancing has been added inside DataNode.

Refactoring daemon and task pair management

2.Maven deployment

Blog

2.1 decompression

[root@hadoop1 softwore] # pwd

/ opt/softwore

[root@hadoop1 softwore] # ls

Apache-maven-3.3.9-bin.zip hadoop-2.8.1-src.tar.gz jdk-8u45-linux-x64.gz

Findbugs-1.3.9.zip hadoop-2.8.1.tar.gz protobuf-2.5.0.tar.gz

[root@hadoop1 softwore] # unzip apache-maven-3.3.9-bin.zip

2.2 configure the mavne directory

2.3 View the configuration files and extract the warehouse files we have prepared

3.Hadoop compilation

3.1 decompression

3.2 View pom.xml

3.3View BUILDING.txt

Requirements: compiling software environment requirements

* Unix System

* JDK 1.7 +

* Maven 3.0 or later

* Findbugs 1.3.9 (if running findbugs)

* ProtocolBuffer 2.5.0

* CMake 2.6or newer (if compiling native code), must be 3.0or newer on Mac

* Zlib devel (if compiling native code)

* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)

* Linux FUSE (Filesystem in Userspace) version 2.6or above (if compiling fuse_dfs)

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

3.4 JDK deployment

[root@hadoop1 softwore] # tar-zxvf jdk-8u45-linux-x64.gz-C / usr/java [root@hadoop1 softwore] # ls-ld / usr/java/*

Drwxr-xr-x 8 root root 4096 December 15 2016 / usr/java/djdk1.7.0_79

Drwxr-xr-x 8 uucp 143 4096 April 11 2015 / usr/java/jdk1.8.0_45

Drwxr-xr-x 8 uucp 143 4096 October 7 2015 / usr/java/jdk1.8.0_65

[root@hadoop1 softwore] # vim / etc/profile

Export JAVA_HOME=/usr/java/jdk1.8.0_45

Export JRE_HOME=/usr/java/jdk1.8.0_45/jre

[root@hadoop1 softwore] # source / etc/profile

[root@hadoop1 softwore] # java-version

Java version "1.8.045"

Java (TM) SE Runtime Environment (build 1.8.0_45-b14)

Java HotSpot (TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

3.5 MAVEN

[root@hadoop000 hadoop-2.8.1-src] # mvn-- version

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)

Maven home: / opt/software/apache-maven-3.3.9

Java version: 1.8.0_45, vendor: Oracle Corporation

Java home: / usr/java/jdk1.8.0_45/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"

[root@hadoop000 hadoop-2.8.1-src] #

3.6 FINDBUGS

[root@hadoop000 findbugs-1.3.9] # findbugs- version

1.3.9

[root@hadoop000 findbugs-1.3.9] #

3.7 PROTOCBUF

[root@hadoop000 local] # protoc-- version

Libprotoc 2.5.0

[root@hadoop000 local] #

3.8 OTHER

3.9 compilation

Mvn clean package-Pdist,native-DskipTests-Dtar

3.10 interpretation

4.hadoop deployment

The stand-alone deployment process does not

Pseudo-distributed deployment process exists + 1 node development

Cluster deployment process exists + n node development / production

Downloaded package: src source code package does not contain jar small

Large components compiled without src or bin

4.1 decompression

Tar-xzvf hadoop-2.8.1.tar.gz

Chown-R root:root hadoop-2.8.1

4.2 interpretation of decompressed files

[root@hadoop000 hadoop-2.8.1] # ll

Total 148

Drwxrwxr-x. 2 root root 4096 Jun 2 2017 bin

Drwxrwxr-x. 3 root root 4096 Jun 2 2017 etc

Drwxrwxr-x. 2 root root 4096 Jun 2 2017 include

Drwxrwxr-x. 3 root root 4096 Jun 2 2017 lib

Drwxrwxr-x. 2 root root 4096 Jun 2 2017 libexec

-rw-rw-r--. 1 root root 99253 Jun 2 2017 LICENSE.txt

-rw-rw-r--. 1 root root 15915 Jun 2 2017 NOTICE.txt

-rw-r--r--. 1 root root 1366 Jun 2 2017 README.txt

Drwxrwxr-x. 2 root root 4096 Jun 2 2017 sbin

Drwxrwxr-x. 4 root root 4096 Jun 2 2017 share

[root@hadoop000 hadoop-2.8.1] #

Bin shell to execute the command

Etc profile

Lib library

Sbin starts and shuts down hadoop

Share jar

[root@hadoop000 hadoop-2.8.1] # rm-f bin/*.cmd

[root@hadoop000 hadoop-2.8.1] # rm-f sbin/*.cmd

[root@hadoop000 hadoop-2.8.1] #

[root@hadoop000 hadoop-2.8.1] # ll bin

Total 348

-rwxrwxr-x. 1 root root 139387 Jun 2 2017 container-executor

-rwxrwxr-x. 1 root root 6514 Jun 2 2017 hadoop

-rwxrwxr-x. 1 root root 12330 Jun 2 2017 hdfs

-rwxrwxr-x. 1 root root 6237 Jun 2 2017 mapred

-rwxrwxr-x. 1 root root 1776 Jun 2 2017 rcc

-rwxrwxr-x. 1 root root 156812 Jun 2 2017 test-container-executor

-rwxrwxr-x. 1 root root 14416 Jun 2 2017 yarn

[root@hadoop000 hadoop-2.8.1] #

4.3 configure environment variables

[root@hadoop000 ~] # vi / etc/profile

Export HADOOP_HOME=/opt/software/hadoop-2.8.1

Export PATH=$HADOOP_HOME/bin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MVN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop000 ~] # source / etc/profile

[root@hadoop000 ~] # which hadoop

/ opt/software/hadoop-2.8.1/bin/hadoop

4.4 configure the core-site file

Etc/hadoop/core-site.xml:

Fs.defaultFS

Hdfs://localhost:9000

Etc/hadoop/hdfs-site.xml:

Dfs.replication

one

4.5ssh

[root@hadoop000 ~] # rm-rf .ssh

[root@hadoop000 ~] # ssh-keygen

[root@hadoop000 ~] # cd .ssh

[root@hadoop000 .ssh] # ll

Total 8

-rw-. 1 root root 1671 May 13 21:47 id_rsa

-rw-r--r--. 1 root root 396 May 13 21:47 id_rsa.pub

[root@hadoop000 .ssh] # cat id_rsa.pub > > authorized_keys

[root@hadoop000 .ssh] #

[root@hadoop000 ~] # ssh localhost date

The authenticity of host 'localhost (:: 1)' can't be established.

RSA key fingerprint is ec:85:86:32:22:94:d1:a9:f2:0b:c5:12:3f:ba:e2:61.

Are you sure you want to continue connecting (yes/no)? Yes

Warning: Permanently added 'localhost' (RSA) to the list of known hosts.

Sun May 13 21:49:14 CST 2018

[root@hadoop000 ~] #

[root@hadoop000 ~] #

[root@hadoop000 ~] # ssh localhost date

Sun May 13 21:49:17 CST 2018

[root@hadoop000 ~] #

4.6 Format the filesystem:

$bin/hdfs namenode-format

4.7 java home configuration

[root@hadoop000 hadoop] # vi hadoop-env.sh

Export JAVA_HOME=/usr/java/jdk1.8.0_45

4.8 Start NameNode daemon and DataNode daemon:

[root@hadoop000 hadoop-2.8.1] # sbin/start-dfs.sh

Starting namenodes on [localhost]

Localhost: starting namenode, logging to / opt/software/hadoop-2.8.1/logs/hadoop-root-namenode-hadoop000.out

Localhost: starting datanode, logging to / opt/software/hadoop-2.8.1/logs/hadoop-root-datanode-hadoop000.out

Starting secondary namenodes [0.0.0.0]

The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

RSA key fingerprint is ec:85:86:32:22:94:d1:a9:f2:0b:c5:12:3f:ba:e2:61.

Are you sure you want to continue connecting (yes/no)? Yes

0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.

0.0.0.0: starting secondarynamenode, logging to / opt/software/hadoop-2.8.1/logs/hadoop-root-secondarynamenode-hadoop000.out

[root@hadoop000 hadoop-2.8.1] #

[root@hadoop000 hadoop-2.8.1] #

[root@hadoop000 hadoop-2.8.1] #

[root@hadoop000 hadoop-2.8.1] #

[root@hadoop000 hadoop-2.8.1] # jps

16243 Jps

15943 DataNode

5127 Launcher

16139 SecondaryNameNode

15853 NameNode

[root@hadoop000 ~] # hdfs dfs-put jepson.log /

[root@hadoop000 ~] #

[root@hadoop000 ~] #

[root@hadoop000 ~] # hdfs dfs-ls /

Found 1 items

-rw-r--r-- 3 root supergroup 6 2018-05-13 21:57 / jepson.log

[root@hadoop000 ~] #

[root@hadoop000 ~] # hdfs dfs-cat / jepson.log

A

five

six

[root@hadoop000 ~] #

[root@hadoop000 ~] #

[root@hadoop000 ~] # cat jepson.log

A

five

six

[root@hadoop000 ~] #

0513 assignment:

1.hadoop compilation

2.hdfs pseudo-distributed deployment

3. The above two blog articles

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.