Shulou(Shulou.com)06/02 Report--

This article mainly shows you "how to deploy and test Hadoop0.20.0". The content is simple and clear. I hope it can help you solve your doubts. Let me lead you to study and learn this article "how to deploy and test Hadoop0.20.0".

Hadoop0.20.0 deployment and testing

Recently, we have learned a little from the deployment of Hadoop. This article summarizes the main process of deployment and the problems encountered, many of which also learn from the experience of other experts on the network (most of the materials are 0.19 or earlier, and 0.20 has been partially adjusted in configuration).

Hadoop official (it is strongly recommended to read the official original document carefully)

The whole deployment process is mainly divided into the following parts (take * * 0.20.0 version as an example):

The system environment is ubuntu.04, JDK1.6

1. download

Download the Hadoop*** version from any mirror site of Apache

There is a manual under docs in the compressed package and a manual in Chinese under docs/cn.

two。 Preparation before starting the hadoop cluster

2.1 install JDK if the machine is multicore, it is recommended to install JDK1.6, which can improve the efficiency of the program.

Set the environment variables required by Hadoop in the hadoop_env.sh under / home/ [$hadoop_home] / conf, where JAVA_HOME is the required variable.

ExportJAVA_HOME=/usr/lib/jvm/java-6-sun

ExportJAVA_HOME=/usr/lib/jvm/java-6-sun

2.2 configure SSH (for configure cluster)

Install SSH

Take UbuntuLinux as an example:

$sudoapt-getinstallssh

$sudoapt-getinstallssh

Generate key pair

$ssh-keygen-trsa-P''

$ssh-keygen-trsa-P''

Enter directly and use the default path when asked to save the path.

The generated key pair id_rsa,id_rsa.pub is stored by default in the / home/ [$user] / .ssh directory.

Then copy the contents of the id_rsa.pub to the / home/ [$user] / .ssh/authorized_keys file on each machine (including the native machine).

If the authorized_keys file already exists on the machine, add the contents of the id_rsa.pub at the end of the file.

If you don't have the file authorized_keys, just go to cp or scp.

Without authorized_keys

$cpid_rsa.pubauthorized_keys

$cpid_rsa.pubauthorized_keys

Or use the append command

$catid_rsa.pub > > authorized_keys

$catid_rsa.pub > > authorized_keys

In the Hadoop0.20.0 deployment, you must do the following or SSH will not work properly. Change the permissions of the authorized_keys file to be read-write for the current user and read-only for other users.

$chmod600authorized_keys

$chmod600authorized_keys

To further enhance security, you can modify the native file / etc/ssh/sshd_config so that after setting it, SSH can only log in through key authentication.

# remove password authentication

PasswordAuthenticationno

AuthorizedKeyFile.ssh/authorized_keys

# remove password authentication

PasswordAuthenticationno

AuthorizedKeyFile.ssh/authorized_keys .

These are all the contents of the article "how to deploy and Test Hadoop0.20.0". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.