Shulou(Shulou.com)06/03 Report--

I. pre-conditions

The operation of Hadoop depends on JDK, which needs to be pre-installed. For installation steps, please see:

Installation of JDK under Linux II. Configure secret-free login

Communication between Hadoop components needs to be based on SSH.

2.1 configuration Mappin

Configure ip address and hostname mapping:

Add 192.168.43.202 hadoop0012.2 at the end of vim / etc/hosts# file to generate public and private keys

Execute the following command line to generate public and private keys:

Ssh-keygen-t rsa3.3 authorization

Go to the ~ / .ssh directory, view the generated public key and private key, and write the public key to the license file:

[root@@hadoop001 sbin] # cd ~ / .ssh [root@@hadoop001 .ssh] # ll-rw-. 1 root root 1675 March 15 09:48 id_rsa-rw-r--r--. 1 root root 388 March 15 09:48 id_rsa.pub# writes a public key to the license file [root@hadoop001 .ssh] # cat id_rsa.pub > > authorized_ Keys [root @ hadoop001 .ssh] # chmod 600 authorized_ Keys III, Hadoop (HDFS) environment build 3.1 download and extract

Download the Hadoop installation package, here I download the CDH version, download address: http://archive.cloudera.com/cdh6/cdh/5/

# decompress tar-zvxf hadoop-2.6.0-cdh6.15.2.tar.gz 3.2Configuring environment variable # vi / etc/profile

Configure environment variables:

Export HADOOP_HOME=/usr/app/hadoop-2.6.0-cdh6.15.2export PATH=$ {HADOOP_HOME} / bin:$PATH

Execute the source command so that the configured environment variables take effect immediately:

# source / etc/profile3.3 modify Hadoop configuration

Go to the ${HADOOP_HOME} / etc/hadoop/ directory and modify the following configuration:

1. Hadoop-env.sh# JDK installation path export JAVA_HOME=/usr/java/jdk1.8.0_201/2. Core-site.xml fs.defaultFS hdfs://hadoop001:8020 hadoop.tmp.dir / home/hadoop/tmp 3. Hdfs-site.xml

Specify the copy factor and temporary file storage location:

Dfs.replication 1 4. Slaves

Configure the hostname or IP address of all slave nodes. Since it is a stand-alone version, you can specify this machine:

Hadoop0013.4, turn off the firewall

Failure to turn off the firewall may result in inaccessible Hadoop's Web UI interface:

# View firewall status sudo firewall-cmd-state# disable firewall: sudo systemctl stop firewalld.service3.5 initialization

Initialization is required when starting Hadoop for the first time. Go to the ${HADOOP_HOME} / bin/ directory and execute the following command:

[root@hadoop001 bin] #. / hdfs namenode-format3.6 starts HDFS

Go to the ${HADOOP_HOME} / sbin/ directory and start HDFS:

[root@hadoop001 sbin] #. / start-dfs.sh3.7 verify whether the startup is successful

Method 1: execute jps to check whether the NameNode and DataNode services have been started:

[root@hadoop001 hadoop-2.6.0-cdh6.15.2] # jps9137 DataNode9026 NameNode9390 SecondaryNameNode

Method 2: check the Web UI interface, port 50070:

IV. Build Hadoop (YARN) environment 4.1 modify configuration

Go to the ${HADOOP_HOME} / etc/hadoop/ directory and modify the following configuration:

1. Mapred-site.xml# if there is no mapred-site.xml, copy a sample file and then modify cp mapred-site.xml.template mapred-site.xml mapreduce.framework.name yarn 2. Yarn-site.xml yarn.nodemanager.aux-services mapreduce_shuffle 4.2 to start the service

Go to the ${HADOOP_HOME} / sbin/ directory and start YARN:

. / start-yarn.sh4.3 verifies whether the startup is successful

Method 1: execute the jps command to see if the NodeManager and ResourceManager services have been started:

[root@hadoop001 hadoop-2.6.0-cdh6.15.2] # jps9137 DataNode9026 NameNode12294 NodeManager12185 ResourceManager9390 SecondaryNameNode

Method 2: check the Web UI interface. The port number is 8088:

For more articles in big data's series, please see the GitHub Open Source Project: big data's getting started Guide.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.