Shulou(Shulou.com)06/03 Report--

Please read the installation of linux in the previous article.

Environment: centos6.5 3 hosts (replication host, can directly copy virtual machine files) one master, replication is slave.

Master-ip:192.168.237.10 slave1-ip:192.168.237.11 slave2-ip:192.168.237.12

Online debugging: enter the ifcfg-eth0 file in slave1 and slave2 and modify the IPADDR to 11 and 12 respectively.

Enter secureCRT and add the connection ports of slave1 and slave2 (enter the hostname and username attributes under ssh3)

Then enter the three mainframes. Test whether you can surf the Internet. (if the copy doesn't work. You can delete the network card and add the trial ^ _ ^)

Software package: jdk-6u45-linux-x64.bin hadoop-1.2.1-bin.tar.gz

Step 1: install jdk and configure the java runtime environment

1. You can use the rz command to upload jdk-6u45-linux-x64.bin

two。 You can set it in wmware (right-click the title-- Settings-- File sharing open and specify the shared file location)

Copy the program to the / usr/local/src directory

# cp jdk-6u45-linux-x64.bin / usr/local/src/

# ll jdk-6u45-linux-x64.bin (check permissions to see if you have X permissions)

#. / jdk-6u45-linux-x64.bin (install jdk)

3. Configure java

# vim ~ / .bashrc

Export JAVA_HOME=/usr/local/src/jdk1.6.0_45

Export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/bin

Export PATH=.:$PATH:$JAVA_HOME/bin

# source ~ / .bashrc (to make the configuration take effect)

4. Test java

# java in any directory (if the command is valid, it means success. Or go back to the first step. Check)

Let's configure the java environment for the other two slave

# scp-rp / usr/local/src/jdk1.6.0_45 192.168.237.11:/usr/local/src

# scp-rp / usr/local/src/jdk1.6.0_45 192.168.237.12:/usr/local/src

(here the option-r: copy the entire directory recursively. -p: retain the modification time, access time and access rights of the original file.)

Then modify the environment variable setting ~ / .bashrc file as above.

Step 2: parsing and configuring hadoop-related profile information

An overview:

(1) Hadoop MapReduce adopts Master/Slave structure.

* Master: the only global manager of the entire cluster, with functions such as job management, status monitoring and task scheduling, that is, JobTracker in MapReduce.

* Slave: responsible for the execution of tasks and the return of task status, that is, TaskTracker in MapReduce.

2. JobTracker analysis:

(1) Overview: JobTracker is a backend service process. After starting, it will always monitor and receive heartbeat information from various TaskTracker, including resource usage and task running information.

(2) the main functions of JobTracker:

1. Job control: in hadoop, each application is represented as a job, and each job is divided into multiple tasks. The job control module of JobTracker is responsible for job decomposition and status monitoring.

* the most important thing is status monitoring, including TaskTracker status monitoring, job status monitoring and task status monitoring. Main role: fault tolerance and provide decision-making basis for task scheduling.

two。 Resource management.

3. TaskTracker analysis:

(1) Overview of TaskTracker: TaskTracker is the bridge between JobTracker and Task: on the one hand, it receives and executes various commands from JobTracker: running tasks, submitting tasks, killing tasks, etc.; on the other hand, it periodically reports the status of each task on the local node to JobTracker through the heartbeat. RPC protocol is used to communicate between TaskTracker, JobTracker and Task.

(2) functions of TaskTracker:

1. Reporting heartbeat: Tracker periodically reports all kinds of information on all nodes to JobTracker through heartbeat mechanism. This information consists of two parts:

* Machine-level information: node health, resource usage, etc.

* Task level information: task execution progress, task running status, etc.

two。 Execute commands: JobTracker will issue various commands to TaskTracker, including: start task (LaunchTaskAction), submit task (CommitTaskAction), kill task (KillTaskAction), kill job (KillJobAction) and reinitialize (TaskTrackerReinitAction).

There are two kinds of nodes in the hadoop cluster, one is namenode, and the other is datanode.

Among them, datanode is mainly responsible for data storage, and namenode is mainly responsible for three functions, namely (1) managing metadata (2) maintaining directory tree (3) responding to customer requests

Installation configuration of hadoop: also copy hadoop-1.2.1-bin.tar.gz to / usr/local/src/

Decompress: # tar zxvf hadoop-1.2.1-bin.tar.gz

Enter hadoop-1.2.1 to create a temporary file tmp for hadoop to run

# cd hadoop-1.2.1 & & mkdir tmp

Modify the configuration file:

1. # vim master

Master

2. # vim slaves

Slave1

Slave2

3. # vim core-site.xml (set tmp directory and namenode node)

# add the following

Hadoop.tmp.dir

/ usr/local/src/hadoop-1.2.1/tmp

Fs.default.name

Hdfs://192.168.237.10:9000

4.#vim mapred-site.xml (configure jobtracker)

# add the following

Mapred.job.tracker

Http://192.168.237.10:9001

5. # vim hdfs-site.xml

# add the following

Dfs.replication

three

6.# vim hadoop-env.sh

Add: export JAVA_HOME=/usr/local/src/jdk1.6.0_45 at the end

# source hadoop-env.sh

Check to see if the modification of the configuration file is complete.

Then copy it to slave1 and slave2.

# scp-rp hadoop-1.2.1 slave1:/usr/local/src/ (the slave1 here is used after the 192.168.237.11 machine is configured with hostname.)

# scp-rp hadoop-1.2.1 slave2:/usr/local/src/ (if hostname is not configured, use the ip address)

Configure DNS and hosts for master

# vim / etc/hosts

# add the following

192.168.237.10 master

192.168.237.11 slave1

192.168.237.12 slave2

# vim / etc/sysconfig/network

HOSTNAME=master

# hostname master

Modifications to slave1 and slave2:

# vim / etc/hosts

192.168.237.10 master

192.168.237.11 slave1

192.168.237.12 slave2

Slave1:

# vim / etc/sysconfig/network

HOSTNAME=slave1

# hostname slave1

Slave2:

# vim / etc/sysconfig/network

HOSTNAME=slave2

# hostname slave2

Step 3: password-free login between hosts is required below; configure openssl

It is recommended to close the protective wall and selinux first.

# / etc/init.d/iptable stop

# setenforce 0

Enter the master host:

# ssh-keygen (then enter all the way)

# cd ~ / .ssh/ (check whether a public key file is generated)

# cat id_rsa.pub > authorized_keys (authorized_keys is an authentication file that stores the public key. It is a common need of all three hosts)

Enter the slave1 host and then the slave2 host:

# ssh-keygen

# cd ~ / .ssh/

Copy the contents of the id_rsa.pub file of slave1 and slave2 to the authorized_keys in master respectively.

Then go to distribute ~ / .ssh/authorized_keys in master to ~ / .ssh/ in slave1 and slave2.

Step 4: start hadoop

#. / hadoop namenode-format (initialize namenode)

#. / start-all.sh (start hadoop)

# jsp (View Information)

If there is any exception, please check the configuration file.

Here are a few simple hadoop commands

# cd / usr/local/src/hadoop-1.2.1/bin

#. / hadoop fs-ls /

#. / hadoop fs-put / usr/local/src/hadoop-1.2.1/README.txt /

#. / hadoop fs-ls /

-rw-r--r-- 3 root supergroup 1366 2018-03-26 18:57 / README.txt

If you think you can, please continue to follow. There will be series and new ones one after another.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.