Shulou(Shulou.com)06/03 Report--

Refer to Liu Peng's book "Practical Hadoop" and follow hadoop 0.20.2 for several points of attention.

First, understand several background processes in Hadoop.

NameNode, Secondary NameNode, JobTracker,TaskTracker,DataNode.

NameNode: Responsible for how to segment data blocks and which nodes to place after segmentation. It centrally manages memory and I/O.

This process is deployed on the Master node, which is a single point, and it hangs the entire system.

Secondary NameNode: Like NameNode, auxiliary program. Each cluster has one that communicates with NameNode, periodically saves HDFS metadata snapshots, and can be used as a backup NameNode in case of NameNode failure. It is also deployed on the Master node.

JobTracker schedules jobs, determines which files are run by which nodes, and listens for heartbeats sent by TaskTracker. When no heartbeat is received, it is considered that a task has failed and will decide to restart the task. There is only one JobTracker per cluster. It is deployed on the Master node.

The three processes mentioned above are deployed on the Master node, while the TaskTracker and DataNode processes need to be deployed at several points in the cluster.

DataNode is responsible for reading and writing HDFS data blocks to the local file system. When the client reads or writes a database, the NameNode tells the client to go to that DataNode, and then the client communicates directly with the server of this DataNode and operates on the relevant data blocks.

TaskTracker is also located in the slave node, it is responsible for independent execution of specific tasks, each slave node can only have one TaskTracker, but each TaskTracker can generate multiple Java virtual machines, used to process multiple maps and reduce ideas in parallel. TaskTracker also interacts with JobTracker, which is responsible for assigning Tasks and detecting TaskTracker heartbeats. If there is no heartbeat, it is considered to have crashed and will be assigned to other TaskTracker.

The deployment diagram for each process is as follows:

Specific installation links, you can refer to the steps in the book, but there are several points to note.

The host and slave create a special user grid running hadoop, and set the password-free login mechanism of SSH. Please refer to http://chenlb.iteye.com/blog/211809. Integrate the contents of the public key files on all machines into an authorized_keys file, so as to realize mutual password free login ssh.

When starting hadoop, note that you should log in as grid user and operate under the home directory of grid user. Sometimes there is permission problem. At this time, you should pay attention to setting the owner of hadoop folder of host and slave to grid user and group. Run chown -R grid:grid /home/grid/hadoop-1.2.1 (here is the directory where hadoop is placed, here you need to use root to modify)

Then you can go to the bin directory in the hadoop folder to start start-all.sh, you can see the following message, indicating that the startup was successful.

At this time, you can also check the startup of the process by running the command. Run the jps file in jdk on the host. You can see the following:

Running the same command on the slave node, you can see

The installation of Hadoop has been successful.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.