Shulou(Shulou.com)06/03 Report--

First, configure three virtual machines

1. Network configuration

Virtualization software: Vmware Workstations 10.7z

Linux system: Centos 6.5-x86_64

Use 4 machines with Linux environment installed to build a small distributed cluster.

Cluster machine details

1. Virtualized network configuration: bridging network

2.IP address is set to fixed IP address

3. Hostname configuration, which needs to be modified

(1) configure vi / etc/hostname

(2) configure vi / etc/sysconfig/network

(3) configure vi / etc/hosts

4. Turn off the firewall

1) take effect after restart

Enable: chkconfig iptables on

Turn off: chkconfig iptables off

2) effective immediately and expire after restart

Enable: service iptables start

Turn off: service iptables stop

The effect after the configuration is completed:

The three nodes communicate with each other by ping

The hostname can be resolved normally

You can connect to the extranet with normal ping.

2. JAVA system environment configuration

Installation of Java

Rpm-qa | grep java to view the java installed on the system

Uninstall the installed java

Rpm-e-nodeps (package name)

Configure the environment variable vi / etc/profile

# set java environment

Export JAVA_HOME=/usr/java/jdk1.8.0_77

Export JRE_HOME=/usr/java/jdk1.8.0_77/jre

Export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

Export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

After the configuration is completed, the effect is:

3. Password-less login between nodes

SSH settings require different operations on the cluster, such as start, stop, and distributed daemon shell operations. Authenticating different Hadoop users requires a public / private key pair provided by Hadoop users and shared by different users.

The following command is used to generate a value pair that uses the SSH key. Copy the public key to form an id_rsa.pub into the authorized_keys file, and provide the owner with read and write access to the authorized_keys file.

Command: ssh-keygen-t rsa

Command: cat ~ / .ssh/id_rsa.pub > > ~ / .ssh/authorized_keys

After the same command on other nodes, add the id_rsa.pub content of all nodes to the authorized_keys, and then distribute the authorized_keys to the /. Ssh/ directory of all nodes

After the configuration is completed, the effect is:

II. Hadoop configuration and installation

The Hadoop version of this document is: hadoop-2.6.4.tar.gz

1. Configure the following on the Namenode node

After downloading Hadoop, you can operate the Hadoop cluster in one of the following three support modes:

Local / stand-alone mode: download Hadoop on the system, after which, by default, it will be configured in a separate mode for running Java programs.

Simulation distributed mode: this is a distributed simulation on a single machine. Hadoop daemon each process, such as hdfs, yarn, MapReduce, etc., will run as a separate java program. This pattern is very useful for development.

Fully distributed mode: this mode is a fully distributed cluster of at least two or more computers. We use this model in future chapters.

(1) change the file core-site.xml to the following configuration:

Fs.default.name

Hdfs://192.168.1.220:9000

Hadoop.proxyuser.root.hosts

192.168.1.220

Hadoop.proxyuser.root.groups

*

(2) change the file hdfs-site.xml to the following configuration:

Dfs.name.dir

/ usr/hadoop/name

Dfs.data.dir

/ usr/hadoop/data

Dfs.replication

two

Dfs.support.append

True

Dfs.datanode.address

0.0.0.0:50110

(3) change the file mapred-site.xml to the following configuration:

Mapred.job.tracker

Http://192.168.1.220:9001

Mapried.local.dir

/ usr/hadoop/var

(4) change the file slaves to the following configuration:

Master-Hadoop

Slave1-Hadoop

Slave2-Hadoop

(5) change the file hadoop-env.sh to the following configuration:

# The java implementation to use.

Export JAVA_HOME=/usr/java/jdk1.8.0_77

two。 Innovate the folder in the configuration under / usr/hadooop/

Mkdir tmp

Mkdir logs

Mkdir data

3.hadoop command operation

(1) use the command hdfs namenode-format to set the name node as follows

(2) start the command / usr/Hadoop/etc/Hadoop/start-all.sh

(3) check node status Hadoop dfsadmin-report

4. Web page node display

Http://Master-Hadoop:50070

Http:// Master-Hadoop:8088

Reference website:

Http://itindex.net/detail/46949-wordcount

Http://www.cnblogs.com/scotoma/archive/2012/09/18/2689902.html

Http://dblab.xmu.edu.cn/blog/install-hadoop-cluster/

Http://192.168.1.200:50070/dfshealth.html#tab-datanode

Http://www.tuicool.com/articles/veim6bU

Http://my.oschina.net/u/570654/blog/112780

Http://blog.csdn.net/ab198604/article/details/8271860

Http://www.cnblogs.com/shishanyuan/category/709023.html

Http://zhidao.baidu.com/link?url=K6w-swVrs7VtvcG8iF1UgQ4dBk3e7YCHoRwA3kjcA3D4fiE2RYdKQNN5h9NJCgg7z5HEvyu7Gyt1jLe7-UMvmGbRoISDTIN6MkcVPM2vbOa

Http://blog.csdn.net/tralonzhang/article/details/7773824

Http://www.cnblogs.com/shishanyuan/p/4177908.html

Http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_user_guide.html

The list of files is:

CentOS-6.5-x86_64-bin-DVD1.iso

Hadoop-2.6.4.tar.gz

Jdk-8u77-linux-x64.rpm

Putty_V0.63.0.0.43510830.exe

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.