Shulou(Shulou.com)06/01 Report--

This article mainly introduces "the method of building hadoop2.7.1 environment". In the daily operation, I believe that many people have doubts about the method of building hadoop2.7.1 environment. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "how to build hadoop2.7.1 environment". Next, please follow the editor to study!

With the support of the boss, he successively pulled to more than 10 machines, bound to a fixed IP, and started a trip to Hadoop after work. It will be recorded for reference.

Hardware composition:

Resource

Volume

CPU

2 cores

Memory

4 GB

Disk

500 GB

Network

100 M

Software composition:

Name

Version

Install Path

CentOS

6.7 x86_64

/

Oracle JDK

7u79-linux-x64

/ usr/local/java/

Hadoop

2.7.1

/ home/hadoop/

Flume

1.6.0

/ home/flume/

Maven

3.3.9

/ usr/local/maven/

Ant

1.9.6

/ usr/local/ant

MySQL

5.6.21 Community Server

/ home/mysql/

D3.js

V3

Overall architecture:

DFS and Yarn constitute:

The configuration required by each node:

Commands used to debug hadoop nodes:

Sbin/hadoop-daemon.sh start journal

Bin/hdfs namenode format

Sbin/hadoop-daemon.sh start namenode

Bin/hdfs namenode-bootstrapStandby

Sbin/hadoop-daemon.sh start namenode

Bin/hdfs haadmin-transitionToActive nn1

Sbin/hadoop-daemons.sh start datanode

Sbin/yarn-daemon.sh start resourcemanager

Sbin/yarn-daemon.sh start nodemanager

The cluster startup / shutdown command after debugging:

Sbin/start-dfs.sh

Sbin/start-yarn.sh

Bin/hdfs haadmin-transitionToActive nn1

Sbin/stop-dfs.sh

Sbin/stop-yarn.sh

DFS management interface:

YARN management interface:

150G logs are being manually imported into DFS. Flume has not been used yet, and then gradually integrated.

The results of MR operation are stored in DFS or injected into MySQL successfully, and then sorted out later.

Yarn's resource queue is temporarily configured, and now it can only run. I still don't understand what's going on. I take the time to continue my research.

It takes 8 minutes to run a useragent-filtered MR for 15G logs on 4 data node nodes (9 nodes last night). It takes 1 day to parse 1 month's logs. Performance optimization needs to be carried out one after another.

At this point, the study of "how to build a hadoop2.7.1 environment" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.