Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to configure Hadoop distributed cluster under 64-bit linux". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to configure Hadoop distributed cluster under 64-bit linux".

Generally speaking, building a Hadoop consists of the following steps:

Check the cluster sub-environment and prepare the appropriate Hadoop deployment file, which is generally as follows:

Check the Java installation (usually do not use your own openJDK, the specific reason is not clear)

Self-compiling Hadoop 64-bit deployment files (32bit Hadoop deployment files will have some minor problems on x64 machines)

Organize the machine in the distributed environment, list its ip (need to change / ect/hosts file), and divide its Master and Slaver responsibilities (can also be divided into more detailed ResourceManager, SecondNameNode, etc., generally speaking, each machine is fully utilized and acts as a DataNode)

"Planning" Hadoop folder structure, including installation directory, data file (hadoop_data), temporary file (hadoop_tmp), etc. Generally speaking, it is recommended to set up hadoop x.x.0 directory (location of unzipped Hadoop deployment files), hadoop-data (data file) and hadoop_tmp (temporary file) under a hadoop x.x.0 directory. Of course, this is the author's own practice, if it is not "industry standard", please do not laugh.

Configure the runtime environment, which refers to the system-related configuration other than the configuration file in the Hadoop deployment file:

Modify / etc/hosts file

Set ssh password-free access to each Master-Slaver

Configure Hadoop parameters:

Modify the configuration files under {Hadoop_Home} / etc/hadoop/, including hadoop-env.sh, yarn-env.sh, core-site.xml, hdfs-site.xml, yarn-site.xml, mapred-site.xml, and slaves files (there is no masters in hadoop 2.2.0, refer to masters and slaves files).

For the initial installation, the parameters should be configured as little as possible, and then there will be a lot of work on cluster optimization.

The configuration supports lzo (this time it is too obsessive in details and falls on this, which is actually divided into two steps):

Install lzo (because the cluster requires some libraries under the native lzo installation directory)

Install hadoop-lzo (the cluster requires some libraries under hadoop-lzo for Hadoop)

Lzop (optional, if you need to test the effect, you can use this tool to generate some .lzo files)

Start the cluster:

Format NameNode:$ {Hadoop_Home} / bin/hdfs namenode-format

One-click startup: ${Hadoop_Home} / sbin/start-all.sh (this method is not officially recommended, but it still works: This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh)

Check the status of the process: jps (tool from JDK)

Refinement steps:

Start the daemon

[hadoop@hadoop01 ~] $hadoop-daemon.sh start namenode

[hadoop@hadoop01 ~] $hadoop-daemon.sh start datanode

Open the yarn daemon

[hadoop@hadoop01 ~] $yarn-daemon.sh start resourcemanager

[hadoop@hadoop01 ~] $yarn-daemon.sh start nodemanager

[hadoop@hadoop01 ~] $start-yarn.sh

Thank you for your reading. the above is the content of "how to configure Hadoop distributed cluster under 64-bit linux". After the study of this article, I believe you have a deeper understanding of how to configure Hadoop distributed cluster under 64-bit linux. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.