Shulou(Shulou.com)06/03 Report--

When I had nothing to do, I suddenly saw something about Hadoop clusters. I have also learned before that I have found some documents about Baidu, Google and other underlying hadoop clusters on the Internet, but in the face of many unfamiliar technologies, I can't see through them very well. All want to try do-it-yourself virtual machines. Often hear such a high-end noun, Hadoop has become synonymous with big data. In just a few years, Hadoop has changed from an edge technology to a de facto standard. MapReduce, on the other hand, is no longer prominent at Google. When companies focus on MapReduce, Google seems to have entered the next era.

Hadoop supports three startup cluster modes, which are stand-alone mode, wei distributed mode and secure distributed mode. The next test I do is in stand-alone mode.

I use the free trial version of Aliyun ECS host.

Uname-a

Linux iZ25wbw8q4uZ 3.13.0-32-generic # 57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86 "64 GNU/Linux

1. Create hadoop households and groups

Enter will be prompted to enter a new UNIX password, which is the password of the new user hadoop. Enter the enter.

If you do not enter the password, enter will re-prompt for the password, that is, the password cannot be empty.

Finally, confirm whether the information is correct. If there is no problem, enter Y and enter.

Change hadoop to root permission

Vim / etc/sudoers

Modify and add hadoop ALL= (ALL:ALL) ALL

2. Install ssh support service to achieve secret-free login

Query ssh package support and create a ssh secret-free login server

Command used: su hadoop / / switch user

Dpkg-l | grep openssh-server / / query ssh support package

Service ssh start / / start the ssh remote connection service

Ssh-keygen-t rsa / / generate ssh private / public key password file

Cat ~ / .ssh/id_rsa.pub > > ~ / .ssh/authorized_keys / / copy the public key authentication file

Ssh localhost / / Native ssh login test

3. Install the JAVA environment

Http://www.oracle.com/technetwork/java/javase/downloads/index.html download jdk

The java-version command appears as follows because the environment variable is not set correctly

Tar zxvf jdk-8u51-linux-x64.tar.gz

Mv jdk1.8.0_51/ / usr/local/src/jdk1.8

Modify environment variabl

Vim / .bashrc

4, install Hadoop2.7.1

Http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz

Tar zxvf hadoop-2.7.1.tar.gz

Mv hadoop-2.7.1 / usr/local/hadoop

Chown-R hadoop.hadoop hadoop

Chmod 774 / usr/local/hadoop/

Hadoop@iZ25wbw8q4uZ:/usr/local/hadoop$ vim ~ / .bashrc / / modify the hadoop user environment variable, and add it at the bottom

# HADOOP VARIABLES START

Export JAVA_HOME=/usr/local/src/jdk1.8/

Export HADOOP_INSTALL=/usr/local/hadoop

Export PATH=$PATH:$HADOOP_INSTALL/bin

Export PATH=$PATH:$HADOOP_INSTALL/sbin

Export HADOOP_MAPRED_HOME=$HADOOP_INSTALL

Export HADOOP_COMMON_HOME=$HADOOP_INSTALL

Export HADOOP_HDFS_HOME=$HADOOP_INSTALL

Export YARN_HOME=$HADOOP_INSTALL

Export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native

Export HADOOP_OPTS= "- Djava.library.path=$HADOOP_INSTALL/lib"

# HADOOP VARIABLES END

Hadoop@iZ25wbw8q4uZ:/usr/local/hadoop$ vim / usr/local/hadoop/etc/hadoop/hadoop-env.sh / / modify the handoop authentication environment variable and add a line at the bottom

5Cool WordCount test

Installation in stand-alone mode is complete. Verify whether the installation is successful by executing WordCount with hadoop instance.

Create an input folder under / usr/local/hadoop path

Mkdir input

Cp README.txt input / / copy README.txt to input

Execute WordCount

Hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.7.1-sources.jar org.apache.hadoop.examples.WordCount input output

Execute cat output/*, to view character statistics results

At this point, the hadoop stand-alone mode is complete.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.