Shulou(Shulou.com)06/01 Report--

This article mainly shows you "how to install Hadoop in local and pseudo-distributed mode", which is easy to understand and well-organized. I hope it can help you solve your doubts. Let me lead you to study and learn "how to install Hadoop in local and pseudo-distributed mode".

I. preliminary knowledge

A) support platform

L GNU/Linux is supported by both development platform and production platform.

L Windows also supports it (not recommended and not covered in this article).

B) necessary software

I. hadoop was developed by java, so the java environment must be installed.

ii. The sshd service needs to be started, and hadoop manages the remote daemon through ssh.

C) install the software

If your cluster does not have the above-mentioned software installed, you need to install them first.

i. Install java.

ii. Install ssh.

Second, download

Download the hadoop2.4 version.

Prepare to start hadoop

Extract the downloaded hadoop package. Edit the file etc/hadoop/hadoop-env.sh. Add or define the following two parameters:

# set java installation directory export JAVA_HOME=/usr/java/jdk_***# set your hadoop installation directory export HADOOP_PREFIX=/usr/local/hadoop

Verify using the following command:

Bin/hadoop

If the setting is successful, the usage document for the hadoop command is displayed.

Now you can install your hadoop cluster in three ways.

L local mode

L pseudo distribution pattern

L distribution pattern

IV. Local model

The default configuration of Hadoop is local mode, which allows hadoop in a java process. This mode is often used for debugging.

Verify it in the following way:

Mkdir inputcp etc/hadoop/*.xml inputbin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep input output 'dfs [a murz.] +'

It would be right if there were any results in output.

Cat output/*

5. Pseudo-distribution pattern

Hadoop can also be run in a pseudo-distributed manner on a single node. Each daemon for Hadoop runs as a different java process.

A) configuration

Etc/hadoop/core-site.xml:

Fs.defaultFS hdfs://localhost:9000

Etc/hadoop/hdfs-site.xml:

Dfs.replication 1

B) Ssh password-free login

L Ssh password-free login

C) start

The following example is to run the mapreduce program locally. If you want to run mapreduce on yarn, please see the yarn single node installation.

i. Format the file system.

Bin/hdfs namenode-format

ii. Start the namenode daemon and the datanode daemon:

Sbin/start-dfs.sh

The Hadoop daemon log output directory is specified by $HADOOP_LOG_DIR (default is $HADOOP_HOME/logs)

iii. Browse namenode through the browser:

Http://localhost:50070/

iv. Create a folder on hdfs to execute the mapreduce program.

Bin/hdfs dfs-mkdir / userbin/hdfs dfs-mkdir / user/

v. Copy the file to hdfs:

Bin/hdfs dfs-put etc/hadoop input

vi. Run the sample program

Bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep input output 'dfs [a murz.] +'

vii. Check the output file

Bin/hdfs dfs-cat output/*

viii. Close the hdfs daemon.

Sbin/stop-dfs.sh

VI. Yarn single point installation

By setting some parameters and running the ResourceManagers daemon and the NodeManager daemon. We can also run mapreducer tasks in pseudo-distributed mode on yarn.

A) configuration

Etc/hadoop/mapred-side.xml

Mapreduce.framework.name yarn

Etc/hadoop/yarn-site.xml

Yarn.nodemanager.aux-services mapreduce_shuffle

B) start the ResourceManager daemon and the NodeManager daemon

Sbin/start-yarn.sh

C) View ResourceManager through the browser:

Http://localhost:8088/

D) run the mapreduce task.

E) when you finish running the task, close the daemon

Sbin/stop-yarn.sh above is all the content of the article "how to install Hadoop in local and pseudo-distributed mode". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.