Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to build hadoop stand-alone mode in linux. The editor thinks it is very practical, so I share it with you for reference. I hope you can get something after reading this article.

Operation environment

Ubuntu version-ubuntukylin-14.04.2 64-bit version

Hadoop version-Hadoop2.6.4-release

Virtual machine version-VMware Workstation Pro 12.0

Jvm version-above Jdk1.7

Installation step 1. Install ubuntu

When a virtual machine installs ubuntu, after installing a virtual machine, you can choose clone to copy the same clone version directly to save the installation time.

Install ubuntu and set up hadoop users to prepare for subsequent cluster ssh channels. It is not recommended to use root users for a series of operations. Baidu and root can modify system resources remotely. Keep in mind that the specific shell script is as follows

Groupadd hadoopuseradd-mg hadoop hadoop#-m means to create a new folder under the / home/ directory that is the same as the user name #-g means to add the user group passwd hadoop# enter and set the password to 123456 ps: the password is limited to development and debugging, do not be careless in the production environment, and set up a password with high complexity to ensure security. 1. Install jvm

Download the tar.gz package corresponding to Java as shown in the following figure

Download the rpm package under Ps:centos and use the RPM command directly to install it. It is recommended to use the gz package under ubuntu. The author tries to install the rpm plug-in for rpm installation, which cripples the system. The experienced god is welcome to leave a message and guide. Thank you.

Create a new jvm directory under the / usr/lib directory, extract the gz package to the / usr/lib/jvm directory, and rename the java-*** package to the java rename command

Mv java-* java

Open the environment variable preexecution file gedit / etc/profile

Add these lines of environment variables

Export JAVA_HOME=/usr/lib/jvm/javaexport JRE_HOME=$ {JAVA_HOME} / jreexport CLASSPATH=.:$ {JAVA_HOME} / lib:$ {JRE_HOME} / libexport PATH=$ {JAVA_HOME} / bin:$PATH

Restart the system after entering the command or execute source / etc/profile directly to preload script variables

Enter the command java-version to print the version information shown above, indicating that jdk has been installed successfully

2. Debug hadoop stand-alone operation

Enter the hadoop scripting environment configuration file

Gedit / opt/hadoop/hadoop/etc/hadoop/hadoop-env.sh

Change JAVA_HOME to absolute path.

Ps: as to why the default configuration of ${JAVA_HOME} does not work, I have not found a reason yet. Welcome the gods to supplement it.

# configure hadoop ssh password-less login-demand background, due to the hadoop cluster rutting dozens of machines for cluster computing

If key input authentication is carried out, it will increase a lot of workload for operation and maintenance, so public key exchange and comparison login is generally carried out in the community.

Enter the hadoop user home directory / home/hadoop/ to find the .ssh directory and create a new one if not. Please change the folder ownership to be owned by the hadoop user.

Install the ssh service

Sudo apt-get install openssh-serversudo apt-get install openssh-client

If you try to enter the command ssh localhost and enter the password interface, it means that the ssh server and client are installed successfully.

L configure password-free login

Enter the home directory of the hadoop user

Cd / home/hadoop

You can see the .sshwe folder and enter the .ssh folder.

Execute the command to generate ssh login-free public and private keys

Ssh-keygen-t dsa-P''- f ~ / .ssh/id_dsa

This command creates two files, id_dsa and id_dsa.pub, under the .ssh folder, which are SSH's private and public keys.

Now append id_dsa.pub to the public key and execute the following command

Cat ~ / .ssh/id_dsa.pub > > ~ / .ssh/authorized_keys

Now switch to hadoop user login and try to connect remotely using ssh

Did ssh localhost get the following picture?

Represents a successful login-free configuration

Enter the hadoop directory / opt/hadoop/hadoop/

Execute the. / sbin/start-all.sh script to start hadoop

Since both hadoop,nameNode and Jobtracker are not started on a stand-alone machine, how to check whether hadoop is started?

View the HDFS file system

. / bin/hadoop fs-ls /

The above message appears. Congratulations, it means that you have successfully installed hadoop on a stand-alone machine. The Hdfs file structure is just like linux.

This is the end of the article on "how to build hadoop stand-alone mode in linux". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.