Shulou(Shulou.com)06/02 Report--

This article is to share with you about how to implement Hadoop installation for Hadoop applications. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

First, install the hadoop system

1. Add users (all namenode and datanode need to use the same user name, and all information transmission requires user authentication)

Freebsd7test1#adduser

Username:rhinuxdoop

Fullname:

Uid (Leaveemptyfordefault):

Logingroup [rhinuxdoop]:

Logingroupisrhinuxdoop.Inviterhinuxdoopintoothergroups? []:

Loginclass [default]:

Shell (shcshtcshbashrbashnologin) [sh]: bash// Note: * use bash. In fact, the shell environment is automatically specified when the program is running.

Homedirectory[ / home/rhinuxdoop]: / services/rhinuxdoop

The client must have the same user name as the server

2 rsync+ssh Hadoop needs to use Hadoop to synchronize status and data, and it can be accessed without entering a password, so set up key authentication.

A, production key pair (use the added user to operate in this case the user is rhinuxdoop)

[rhinuxdoop@freebsd7test1~] $ssh-keygen-tdsa-P''-f~/.ssh/id_dsa

Defaults to ~ / .ssh /

B, add to the local user

[rhinuxdoop@freebsd7test1~] $cat~/.ssh/id_dsa.pub > > ~ / .ssh/authorized_keys

C, add to other datanode

Import the contents of the .ssh / id_rsa.pub file under the namenoderhinuxdoop user directory into another datanode

[rhinuxdoop@freebsd7test2~] $catid_dsa.pub > > ~ / .ssh/authorized_keys

D, test: log in to namenode with rhinuxdoop to ssh to all datanode (including your own machine in this case)

You should be able to log in without a password

3. Download the hadoop installation software

A, go to http://hadoop.apache.org/core/ to download the * version hadoop is currently 0.16.0

B, extract the hadoop-0.16.0.tar.gz to the user directory / services/rhinuxdoop/

[rhinuxdoop@freebsd7test1~] $tarzxvfhadoop-0.16.0.tar.gz.

4 description of the directory structure of Hadoophouse 0.16.0

A bin store script program

Such as the main control program hadoop

Startup script start-all.sh-- starts all hadoop processes start-dfs.sh-- starts DFS process start-mapred.sh starts MAP/Reduce process

Stop script stop-all.shstop-dfs.shstop-mapred.sh

Brecom conf configuration file directory

The environment profile hadoop-env.sh can configure environment variables, such as JAVA_HOME, which is already the system default.

Default profile hadoop-default.xml this file cannot be modified the parameters that need to be set please modify hadoop-site.xml

Custom configuration file hadoop-site.xml this file can customize the configuration, and the configuration items that appear in this file will be automatically replaced by the configuration items in the default configuration file hadoop-default.xml

Namenode specifies the file masters this file specifies the master (namenode) machine name

Datanode specifies the file slaves this file specifies the list of slave (datanode) machines, with each machine name on a line

Cpgl log directory

Default log storage directory

D, the specific parameters of each configuration file are described and studied in the next version. Let's take a look at the hadoop configuration (freebsd7test1) during Hadoop installation.

5 Hadoop configuration (freebsd7test1):

A, set up namenode storage directory and data directory

[rhinuxdoop@freebsd7test1~] $mkdir-p/services/rhinuxdoop/hadoop-0.16.0/filesystem/name

[rhinuxdoop@freebsd7test1~] $mkdir-p/services/rhinuxdoop/hadoop-0.16.0/filesystem/data

B, configure environment variables, edit. / conf/hadoop-env.sh

= hadoop-env.sh=

ExportJAVA_HOME=/usr/local/jdk1.5.0

# at least set JAVA_HOME, or set the HADOOP_HOME- system directory. HADOOP_ Home defaults to the parent directory of the bin directory.

=

C, configure hadoop-site.xml (for instructions on configurable items and configuration items, please see hadoop-default.xml)

The format of the configuration item is:

Io.file.buffer.size 4096 Thesizeofbufferforuseinsequencefiles. Thesizeofthisbuffershouldprobablybeamultipleofhardware pagesize (4096onIntelx86), anditdetermineshowmuchdatais bufferedduringreadandwriteoperations. Configuration item name configuration item value configuration item description this text is only used to help people understand.

Format namenode

[rhinuxdoop@freebsd7test1~] $hadoop-0.16.0/bin/hadoopnamenode-format

No error is fine. The basic configuration of freebsd7test1 has been completed during Hadoop installation.

6 Hadoop configuration (freebsd7test2 and other datanode)

A, other datanode must keep the same directory structure of namenode, so copy the namenode directory over.

[rhinuxdoop@freebsd7test1~] $scp-rhadoop-0.16.0172.16.27.82:/services/rhinuxdoop/

7, start hadoop

[rhinuxdoop@freebsd7test1~] $hadoop-0.16.0/bin/start-all.sh

The directory and file name of the logging will be told. Logging can basically be set in hadoop-site.xml and set to info by default in hadoop-default.xml.

8, view and test.

A, the system opens the namenode listening ports of dfs and jobtracke listening ports of mapreduce at ports 9000 and 9001, respectively.

B, check the datanode status: [rhinuxdoop@freebsd7test1~] $hadoop-0.16.0/bin/hadoopdfsadmin-report can see the status of each point.

C, deposit the file: [rhinuxdoop@freebsd7test1~] $hadoop-0.16.0/bin/hadoopfs-put/home/xuyunfeng/tzupdater-1.3.3-2007k.ziptzupdater-1.3.3-2007k.zip

/ home/xuyunfeng/tzupdater-1.3.3-2007k.zip in the local system directory is stored in the root directory of the hadoop file system.

D, view the hadoop file system directory [rhinuxdoop@freebsd7test1~] $hadoop-0.16.0/bin/hadoopfs-ls

/ user/rhinuxdoop/diablo-caffe-freebsd6-i386-1.5.0_07-b01.tar.bz2543472742008-02-2810:57rw-r--r--rhinuxdoopsupergroup

/ user/rhinuxdoop/tzupdater-1.3.3-2007k.zip2637152008-02-2816:16rw-r--r--rhinuxdoopsupergroup

E, refer to the manual for more hadoop file system command operations

Follow-up to the test by calling the frecoverable API

G, you can browse NameNode and JobTracker through web. The default port is:

◆ NameNode- http://freebsd7test1.rhinux.com:50070

◆ JobTracker- http://freebsd7test1.rhinux.com:50030

◆ needs to configure hosts on the machine you are browsing.

Thank you for reading! This is the end of this article on "how to implement Hadoop installation of Hadoop applications". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.