Shulou(Shulou.com)06/02 Report--

Recently, I learned to build hadoop. It took me a long time to build it. I would like to write a summary of it. I would like to thank you [data magnifying glass] for his article. It is very good. I built it in accordance with his steps. Here, I perfected it, didn't say it, and started it directly: Ps: because my laptop configuration is slightly poor, only 3 virtual machines have been installed.

1: list of tools

Linux uses CentOS 6.5 here.

Hadoop is used here: hadoop-1.2.1-bin.tar.gz

JDK is used here: jdk-8u172-linux-x64.tar.gz

2: install 3 virtual machines of Linux system through VMware, which will not be introduced here (version 6.5 of CentOS is used here)

After installing the system, the following steps are recommended to operate under the root user

3: configure the network

Configure the IP of the three machines as fixed IP, and set the network to NET mode

Configure fixed IP cd / etc/sysconfig/network-scripts Vi ifcfg-eth0 DEVICE= "eth7" BOOTPROTO= "static" # here is to set the fixed IP mode: static fixed DHCP automatically gets # HWADDR= "00:0C:29:2A:0E:89" IPV6INIT= "yes" NM_CONTROLLED= "yes" ONBOOT= "yes" TYPE= "Ethernet" 8ffaa06d-8057-43a1-bb93-1302400da0b9 "IPADDR=192.168.183.130 # here is the setting of IP GATEWAY=192.168.183.2 # gateway I set up 192.168.183.130 (parent) the other two The environment is set to: 192.168.183.128 (child) 192.168.183.129 (child) set the login method of the virtual machine: (here I personally set the NET connection method Others should also work, you can try) NET networking settings virtual machine (edit -. > virtual network editor), select the subnet IP under NET mode to 192.168.183.0 to enter the NET setting Set the gateway to 192.168.183.2 and then go to the network settings page of your local machine: open Network and sharing Center-> change Adapter Settings-> enter VMware Network Adapter VMnet8 Properties IPv4 Settings default Gateway is set to 192.168.183.2 and then restart the network connection of the virtual machine (the computer icon in the lower right corner of the virtual machine, disconnect the connection) Reconnect) set in the settings of the virtual machine-> hardware-> Network Adapter-> Network connection (select custom, check VMnet8 (NET mode)) restart the virtual machine

4: configure hostname and secret-free mutual access

Configure hostname cd / etc; vi hosts append 192.168.183.130 blue 192.168.183.129 blue2 192.168.183.128 blue3 Note: to modify the operation configuration network file cd / etc/sysconfig for three environments respectively Vi network edits the value of network, modifies it to blue blue2 blue3 to restart the virtual machine, and executes the hostname command, and you will find that hostname has been successfully modified to set up secret-free mutual access to generate the key key. First, establish the directory: mkdir / root/.ssh execute the command ssh-keygen (confirmation is required during the process. Directly enter skip-these confirmation steps are prompted to set the password) after execution, two files will be generated in the / root/.ssh directory. The contents of the files generated by the machines whose id_rsa id_rsa.pub does not work are different: for example: id_rsa.pub: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxxuQrPBqE09NVjfHbquem5abgu7htljA7bNddQsAP/ generated in 192.168.183.130 environment 7wuko7dy6JC5lc1roomoant4TrDSusuch QTEEMt4anIyp3dk2FAP1FrSlyZvioC0oPhcAfXlVUeDVeBoadYoVxUeDVeBoadYoVNUnizxDVeBoadYoVNUnizbpliczgBigBiVSN0mklawrZoufang GkNs0I2EtOL8URx6ZGO5H5pyfy8tZeeCoE2BgMw7nZeeCoE2BgMw7nZZBHcnL1InC38Hpn90cyu5vA9x0NTp1pCGKmLMYMkoWnSqPqP69ZkRA4LqP0ZZPZ6SacyFAa40wTx5Mye5M2EtOL8URx6ZGO5HNX8tZeeCoE2BgMw7nZBHcnL1InC38Hpn90cyu5vA9x0NT0AknpCGKmLYMYMkoWnSqPqP69ZkRA4LqP6ZPzSacyFAa40wTx5M2EtOL8URx6ZGO5H5pyfy8tZeeCoE2BgMw7nZeeCoE2BgMw7nZBZL1InC38Hpn90cyu5vA9x0NT0AknpCGKmLYYMkoWnSqPqP69ZkRA4LqP6ZZPZ6SacyFAa40wTx5Mye5M2EtOL8URx6ZGO5H5pyfy8tZeeCoE2BgMw7nZYBHcnL1InC38Hpn90cyu5vA9x0NT0AknpCGKmLYMYMkoWnSqPqP6ZPZ6SacyFAa40wTx5Mye The parent node executes the command: cat id_rsa.pub > authorized_keys and then appends the id_rsa.pub file of the child node to authorized_keys Finally, the root of the appended content is changed to blue. The final authorized_keys file is as follows: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxxuQrPBqE09NVjfHbquem5abgu7htljA7bNddQsAP/7wuko7dy6JC5lc1+oant4TrDSu+QTEEMt4anIyp3dk2FAP1FrSlyZvioC0oPhcAfXlVUe/WOLaoVsXvXc7BaRx/DVeWBoadYoVN/zb/jrgBiVSN0mklawrZou+GkNs0I2EtOL8URx6ZGO5H5pyfy8tZeebCoE2BgMw7nZBHccnL1InCc38Hpn90cyu5vA9xNT0Akp1npCGKmLYMkoWnCdSqPwwM69ZkRHAo4gL/0ZeyAkPzSa6cyFAQa40wTxq5ye8M2eCGBmjhKhauRgyo8SdN7A9sJ63FXMGCJkzI48aIw== root@blue

Ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzkKeSoDQx9AnAKkBoygiWLH25Pp6GlwjaqqG9LnbCwif5LcUX4N2mfWyXrgpJ3C5PB0r4xq7T776U1Q+VUj43/JEBmlFzpzIiQt0k0yB10mzDVdJwHSWspjQGNkCsiCF9gnuUpcxU5QO0J9gCvJhgfoK6mbVM+xBjF3B5pBaIdnSLZJTqlAlG62SH+kOQQWjkhiFzrxJdfViAcq4Leyg5WP+rBs7tKXysmwnFtpbn6Tok4V+p3VGFqg9/p8OB2HwWokHPV/ZISGbo4G6UpEtYp2hfcLJyo8JjJ1LCDszMyyvIrAZwo/wm6HCNuQU5kG4glghpUrhIEwPqivTeKq4eQ== blue@blue2

Ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAucpajUU3o6XCk6V6JxPT7fgUDJ7Cw/tItDEFB9EMDSoCfua+tARPUdickq6qdqcrzE0JS2sB1Clil4p8nO7DL0HPQwK4ER3CEzXUDB9uoUeuDKNDpRHJFZvvDHs1Lqo+OPMZ6hbWVGh2DE+yFLOKBQ3DtbfGYZhePXryqhh/W2FrI/3LjT1DLSTckGa7ZgqaPh6t4YmktpLGzue3KjWzRAb2BPdKqgK5f67w5RSv4s3ttZbAMOGebg/PNdl9M77oSX7HsMIDkvbkogIZDHV+2LWIyHpaCAvBNDzHQd/67+XlRH1GJummFQjFrl3wJ1epJFl2rPhZjmT+37CCnMUbAQ== blue@blue3

Distribute the authorized_keys file to 2 child nodes:

Execute the command on the parent node:

Scp-r authorized_keys 192.168.183.128:~/.ssh

Scp-r authorized_keys 192.168.183.129:~/.ssh

At the end of this step, you will find that the three environments can access each other without secret.

5: configure and install Hadoop (JDK needs to be installed in the middle, configure environment variables)

First of all, the input configuration of the environment: turn off the firewall command of the three environments: service iptables stop View status: service iptables status appears this prompt: iptables: Firewall is not running. Indicates that the firewall is turned off or iptables-L is executed with the following prompt, which also indicates that the firewall has turned off Chain INPUT (policy ACCEPT)

Target prot opt source destination

Chain FORWARD (policy ACCEPT)

Target prot opt source destination

Chain OUTPUT (policy ACCEPT)

Target prot opt source destination

Install JDK (I use jdk-8u172-linux-x64.tar.gz here): extract the downloaded package to the specified directory: (the directory here is the installation and deployment directory / usr/local/src) tar-zxvf jdk-8u172-linux-x64.tar.gz is decompressed to generate the jdk1.8.0_172 directory under the current directory configuration environment variable: vi / root/.bashrc append to the file: # set java path

Export JAVA_HOME=/usr/local/src/jdk1.8.0_172

Export JRE_HOME=$ {JAVA_HOME} / jre

Export CLASSPATH=.:$ {JAVA_HOME} / lib:$ {JRE_HOME} / lib

Export PATH=$ {JAVA_HOME} / bin/:$PATH

Then let the environment variable take effect.

Source .bashrc

Execute the command: java can see if the environment variable is in effect.

Install JDK in the other two environments and configure the environment variables in the same way.

* * install Hadoop software (I use hadoop-1.2.1-bin.tar.gz here): extract the downloaded package to the specified directory: (the directory here Is the installation and deployment directory is / usr/local/src) unpack tar-zxvf hadoop-1.2.1-bin.tar.gz

Modify the extracted directory hadoop-1.2.1 mv hadoop-1.2.1 hadoop (or not, as long as you make sure that the directory name is consistent with the directory name of the following configuration file, but the author has obsessive-compulsive disorder)

Modify the Hadoop tool configuration file:

Cd / usr/local/src/hadoop/conf

There are 6 modified configuration files.

Respectively

Masters

Slaves

Core-site.xml

Mapred-site.xml

Hdfs-site.xml

Hadoop-env.sh

Start configuration: vi masters modifies the content to: blue + + vi slaves modifies the content into the modified file content in the supplementary configuration of blue2 blue3 + + vi core-site.xml

Hadoop.tmp.dir

/ usr/local/src/hadoop/tmp

Fs.default.name

Hdfs://192.168.183.130:9000

+ +

Vi mapred-site.xml

Supplement the content in configuration

The contents of the modified file

Mapred.job.tracker

Http://192.168.183.130:9001

+ +

Vi hdfs-site.xml

Supplement the content in configuration

The contents of the modified file

Dfs.replication

three

+ +

Vi hadoop-env.sh

Append a line to the configuration file:

Export JAVA_HOME= / usr/local/src/jdk1.8.0_172

+ + +

After configuration, distribute the Hadoop directory to the other two environments:

Scp-rp / usr/local/src/hadoop 192.168.183.128:/usr/local/src

Scp-rp / usr/local/src/hadoop 192.168.183.129:/usr/local/src

Check the firewall status again:

Service iptables status

Start Hadoop

The format of the node needs to be changed before startup (the following operations are performed on the parent node):

Cd hadoop/bin;. / hadoop namenode-format

Output:

18-05-13 06:32:14 INFO namenode.NameNode: STARTUP_MSG:

/ *

STARTUP_MSG: Starting NameNode

STARTUP_MSG: host = blue/192.168.183.130

STARTUP_MSG: args = [- format]

STARTUP_MSG: version = 1.2.1

STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2-r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013

STARTUP_MSG: java = 1.8.0,172

* * /

Re-format filesystem in / usr/local/src/hadoop/tmp/dfs/name? (Y or N) y

Format aborted in / usr/local/src/hadoop/tmp/dfs/name

18-05-13 06:32:20 INFO namenode.NameNode: SHUTDOWN_MSG:

/ *

SHUTDOWN_MSG: Shutting down NameNode at blue/192.168.183.130

* * /

Start Hadoop

. / start-all.sh

Output:

Starting namenode, logging to / usr/local/src/hadoop/libexec/../logs/hadoop-root-namenode-blue.out

Blue3: starting datanode, logging to / usr/local/src/hadoop/libexec/../logs/hadoop-root-datanode-blue3.out

Blue2: starting datanode, logging to / usr/local/src/hadoop/libexec/../logs/hadoop-root-datanode-blue2.out

Blue: starting secondarynamenode, logging to / usr/local/src/hadoop/libexec/../logs/hadoop-root-secondarynamenode-blue.out

Starting jobtracker, logging to / usr/local/src/hadoop/libexec/../logs/hadoop-root-jobtracker-blue.out

Blue2: starting tasktracker, logging to / usr/local/src/hadoop/libexec/../logs/hadoop-root-tasktracker-blue2.out

Blue3: starting tasktracker, logging to / usr/local/src/hadoop/libexec/../logs/hadoop-root-tasktracker-blue3.out

Execute the command:

Jps

Output:

3889 SecondaryNameNode

4194 Jps

3971 JobTracker

3740 NameNode

Execute the command on 2 child nodes:

Jps

Output separately:

3859 DataNode

3932 TaskTracker

4030 Jps

2998 TaskTracker

3096 Jps

2925 DataNode

This means that the build configuration is successful.

If it appears when jps is executed:

Bash:jps:command not found

Indicates the problem with the environment variable, check the environment variable setting

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.