Shulou(Shulou.com)06/03 Report--

Summary on the Construction of Hadoop pseudo-distributed Cluster

I. the required software

VMware15!

CentOS6.5

JDK1.8

Hadoop2.7.3

II. Installation

Note:

Edit the file: enter a to indicate that the file is edited, and finally save the file. The operation is as follows: click the Esc button on the keyboard, then enter the English character, then enter wq, click enter, and complete the file saving.

1. Turn off the firewall and disable SELINUX

(1) permanently shut down the firewall and restart the Linux system

(2) disable SELINUX: modify file parameters to restart Linux to make it effective

(3) check whether the firewall is running, and the figure below shows that it is turned off.

2. Configure the correspondence between hostname and IP addresses

(1) check your own IP

(2) check your own hostname

(3) configuration correspondence

Note: there is a space between IP and hostname

3. Create a user

(1), create a hadoop user

(2) set the password for hadoop users

4. Configure SSH secret-free login

(1) check whether SSH is installed on your system.

Note: if SSH is not installed, you can enter: yum install openssh-server to install it if you have a network.

(2) configure SSH secret-free login for hadoop users

Note: be sure to switch to the hadoop user and enter the "~" directory!

Note: enter ssh-keygen-t rsa and press enter all the time!

Then go to the .ssh directory and do the following

Note: "authorized_keys" must be accurate! * * ``*

Enter the "~" directory to grant appropriate permissions to the .ssh directory and its files

5. JDK installation

(1) create a catalog for storing software

(2) upload and decompress jdk

Note: make sure the permission in the red box is hadoop! If root, you can go to root and change it to hadoop with the command: chown-R hadoop:hadoop jdk-8u211-linux-x64.tar.gz, and then re-enter hadoop to decompress.

(3) create a soft connection for jdk1.8.0_211

(4), configure bashrc file

Make the configuration file effective

(5) verify whether JDK is installed successfully. The following figure indicates the success.

6. Installation and configuration of hadoop

(1) upload and decompress hadoop

Note: make sure the permission in the red box is hadoop! If root, you can go to root and change the command to hadoop: chown-R hadoop:hadoop hadoop-2.7.3.tar.gz, and then re-enter hadoop to decompress.

(2) enter the hadoop configuration file directory to configure the corresponding files.

Configure core-site.xml vi core-site.xml

Note: the arrow should refer to your own hostname

Configure hdfs-site.xml vi hdfs-site.xml

Configure hadoop-env.sh vi hadoop-env.sh

Note: this path is the installation directory of JDK

Configure mapred-site.xml vi mapred-site.xml

Configure yarn-site.xml vi yarn-site.xml

Configure slaves vi slaves

Note: this is your own hostname

(3) create a soft connection for hadoop-2.7.3

(4), configure bashrc file

Make the configuration file effective

(5) create a hadoop-related data directory

(6) format Namenode hadoop namenode-format

Note: the presence of the above 2 messages is a success! If you need to enter a password, check whether your SSH secret-free login is configured correctly!

(7) start hadoop pseudo-distributed cluster

Note: enter yes when the above information appears!

(8) View the startup process of hadoop through the jps command

Note: the occurrence of the above five processes means that the hadoop pseudo-distributed cluster starts successfully

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.