Shulou(Shulou.com)06/01 Report--

This article is about how raspberry pie builds hadoop clusters. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Hadoop is implemented by Java, so running on a raspberry pie is as easy as running on other x86 platforms. First, we need to install JVM that supports raspberry pie. You can choose OpenJDK or Oracle's JDK 8. I personally recommend JDK8, which is slightly faster, but OpenJDK is easier to install.

1. Install Java

Installing OpenJDK is simple, as long as you execute the following command

Pi@raspberrypi ~ $sudo apt-get install openjdk-7-jdkpi@raspberrypi ~ $java-versionjava version "1.7.0x07" OpenJDK Runtime Environment (IcedTea7 2.3.2) (7u7-2.3.2a-1+rpi1) OpenJDK Zero VM (build 22.0-b10, mixed mode)

In addition, we can choose to install JDK 8. 0 for Oracle.

You can get it here: https://jdk8.java.net/fxarmpreview/index.html

Pi@raspberrypi ~ $sudo tar zxvf jdk-8-ea-b36e-linux-arm-hflt-*.tar.gz-C / optpi@raspberrypi ~ $sudo update-alternatives-- install "/ usr/bin/java"java" / opt/jdk1.8.0/bin/java "1 pi@raspberrypi ~ $java-versionjava version" 1.8.0-ea "Java (TM) SE Runtime Environment (build 1.8.0-ea-b36e) Java HotSpot (TM) Client VM (build 25.0-b04, mixed mode)

If you have both installed, use the following command to switch:

Sudo update-alternatives-config java

two。 Add a new hadoop system user

Pi@raspberrypi ~ $sudo addgroup hadooppi@raspberrypi ~ $sudo adduser-- ingroup hadoop hduserpi@raspberrypi ~ $sudo adduser hduser sudo

3. Set up SSH

Pi@raspberrypi ~ $su-hduserhduser@raspberrypi ~ $ssh-keygen-t rsa-P ""

This generates a RSA key that matches an empty password. Hadoop will no longer prompt for a password when communicating with other nodes

Hduser@raspberrypi ~ $cat ~ / .ssh/id_rsa.pub > > ~ / .ssh/authorized_keys

Now set SSH to allow access with the newly generated key

Hduser@raspberrypi ~ $ssh localhost

Now we should be able to log in without a password.

4. Install Hadoop

We can download hadoop from http://www.apache.org/dyn/closer.cgi/hadoop/core

Hduser@raspberrypi ~ $wget http://mirror.catn.com/pub/apache/hadoop/core/hadoop-1.1.2/hadoop-1.1.2.tar.gzhduser@raspberrypi ~ $sudo tar vxzf hadoop-1.1.2.tar.gz-C / usr/localhduser@raspberrypi ~ $cd / usr/localhduser@raspberrypi / usr/local$ sudo mv hadoop-1.1.2 hadoophduser@raspberrypi / usr/local$ sudo chown-R hduser:hadoop hadoop

Now hadoop is installed. Compile the .bashrc file in the home directory and add the following to it

Export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-armhfexport HADOOP_INSTALL=/usr/local/hadoopexport PATH=$PATH:$HADOOP_INSTALL/bin

If you are using oracle's JDK, modify JAVA_HOME accordingly.

Restart the raspberry pie to verify that the installation was successful:

Hduser@raspberrypi ~ $hadoop versionHadoop 1.1.2Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1-r 1440782Compiled by hortonfo on Thu Jan 31 02:03:24 UTC 2013From source with checksum c720ddcf4b926991de7467d253a79b8b

5. Configure Hadoop

Note: the configuration here is the minimum configuration for hadoop single-node mode.

The configuration file is located in "/ usr/local/hadoop/conf/". We need to modify the core-site.xml, hdfs-site.xml and mapred-site.xml files.

Core-site.xml

Hadoop.tmp.dir / fs/hadoop/tmp fs.default.name hdfs://localhost:54310

Mapred-site.xml

Mapred.job.tracker localhost:54311

Hdfs-site.xml

Dfs.replication 1

Oh, it's almost finished. There's still one last step left.

Hduser@raspberrypi ~ $sudo mkdir-p / fs/hadoop/tmphduser@raspberrypi ~ $sudo chown hduser:hadoop / fs/hadoop/tmphduser@raspberrypi ~ $sudo chmod 750 / fs/hadoop/tmphduser@raspberrypi ~ $hadoop namenode-format

Note:

If we choose JDK 8, we need to force DataNode to run in JVM client mode, because JDK 8 does not support server mode. Go to the / usr/local/hadoop/bin directory to edit the hadoop file (please back up first). The steps to modify using nano are as follows: nano hadoop, ctrl-w type "- server" to find. We need to delete the parameter "- server" and then save the exit.

The hadoop single-node system is completed. Here are some useful commands.

1. Jps / / output local VM identifier 2. Start-all.sh / / start all hadoop processes 3. Stop-all.sh / / stop all hadoop processes thank you for reading! This is the end of this article on how to build a hadoop cluster for raspberry pie. I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.