Network Security Internet Technology Development Database Servers Mobile Phone Android Software Apple Software Computer Software News IT Information

In addition to Weibo, there is also WeChat

Please pay attention

WeChat public account

Shulou

One of big data's learning series-Hadoop environment building (stand-alone)

2025-01-29 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Share

Shulou(Shulou.com)06/03 Report--

1. Environment selection 1, server selection

Ali CVM: entry-level (postpaid)

Operating system: linux CentOS 6.8

Cpu:1 kernel

Memory: 1G

Hard disk: 40g

Ip:39.108.77.250

2, configuration selection

JDK:1.8 (jdk-8u144-linux-x64.tar.gz)

Hadoop:2.8.2 (hadoop-2.8.2.tar.gz)

3, download address

Official website address:

JDK:

Http://www.oracle.com/technetwork/java/javase/downloads

Hadopp:

Http://www.apache.org/dyn/closer.cgi/hadoop/common

Baidu Cloud:

Link: http://pan.baidu.com/s/1pLqS4kF password: yb79

Second, the configuration of the server

Before configuring Hadoop, you should do the following configuration

1, change the hostname

Change the hostname first in order to facilitate management.

Enter:

Hostname

View the name of this machine

Then enter:

Vim / etc/sysconfig/network

Modify host name

Change the name of HOSTNAME to the name you want to set

Note: after the host name is changed, reboot will not take effect until it is restarted.

Input

Vim / etc/hosts

Add the host IP and the corresponding host name to do the mapping.

Note: this mapping must be done if the hostname is used in the configuration file!

2. Turn off the firewall

Turn off the firewall to facilitate external access.

The following inputs for CentOS 7 version:

Turn off the firewall

Service iptables stop

Input for versions above CentOS 7:

Systemctl stop firewalld.service3, time settin

Enter:

Date

Check whether the server time is the same, and if not, change it.

Change time command

Date-s' MMDDhhmmYYYY.ss' III, Hadoop environment installation 1, download jdk, hadoop

Put the downloaded jdk and hadoop packages in the home directory

And create new java and hadoop folders

3.1.1 extract the file

Enter:

Tar-xvf jdk-8u144-linux-x64.tar.gz

Tar-xvf hadoop-2.8.2.tar.gz

Extract jdk and hadoop, and move files to java and hadoop files, respectively

And rename the folder to jdk1.8 and hadoop2.8

2Det JDK environment configuration

First enter

Java-version

Check to see if JDK is installed, and if so, but the version is not suitable, uninstall it

3.2.1 profile file changes

Edit / etc/profile file

Enter:

Vim / etc/profile

Overall profile:

Export JAVA_HOME=/home/java/jdk1.8export JRE_HOME=/home/java/jdk1.8/jreexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/libexport PATH=.:$ {JAVA_HOME} / bin:$PATH

Note: the path of JAVA_HOME is the path of JDK

Remember to enter after editing with vim

Source / etc/profile

Make the configuration effective

After configuration, type java-version to view the version information

3Get Hadoop environment configuration 3.3.1 profile file changes

Edit / etc/profile file

Enter:

Vim / etc/profile

Overall profile:

Export HADOOP_HOME=/home/hadoop/hadoop2.8export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport HADOOP_OPTS= "- Djava.library.path=$HADOOP_HOME/lib" export PATH=.:$ {JAVA_HOME} / bin:$ {HADOOP_HOME} / bin:$PATH

Enter:

Source / etc/profile

Make the configuration effective

Modify the configuration file

3.3.2 create a new folder

Before modifying the configuration file, now set up some folders in the root directory.

Enter:

Mkdir / root/hadoop mkdir / root/hadoop/tmp mkdir / root/hadoop/var mkdir / root/hadoop/dfs mkdir / root/hadoop/dfs/name mkdir / root/hadoop/dfs/data

Note: create a new folder under the root directory to prevent it from being deleted inexplicably.

First change to the / home/hadoop/hadoop2.8/etc/hadoop/ directory

3.3.3 modify core-site.xml

Enter:

Vim core-site.xml

Before adding:

Hadoop.tmp.dir / root/hadoop/tmp Abase for other temporary directories. Fs.default.name hdfs://test1:9000

Description: test1 can be replaced with the ip of the host.

3.3.4 modify hadoop-env.sh

Enter:

Vim hadoop-env.sh

Modify ${JAVA_HOME} to your own JDK path

Export JAVA_HOME=$ {JAVA_HOME}

Modified to:

Export JAVA_HOME=/home/java/jdk1.8

3.4.5 modify hdfs-site.xml

Enter:

Vim hdfs-site.xml

Before adding:

Dfs.name.dir / root/hadoop/dfs/name Path on the local filesystem where theNameNode stores the namespace and transactions logs persistently. Dfs.data.dir / root/hadoop/dfs/data Comma separated list of paths on the localfilesystem of a DataNode where it should store its blocks. Dfs.replication 2 dfs.permissions false need not permissions

Description: after dfs.permissions is configured as false, you can generate files on dfs without checking permissions, which is convenient, but you need to prevent accidental deletion, please set it to true, or delete the property node directly, because the default is true.

3.4.6 modify mapred-site.xml

If the file is not mapred-site.xml, copy the mapred-site.xml.template file and rename it to mapred-site.xml.

Enter:

Vim mapred-site.xml

Modify the newly created mapred-site.xml file and add the configuration to the node:

Mapred.job.tracker test1:9001 mapred.local.dir / root/hadoop/var mapreduce.framework.name yarn

Commands used:

At this point, the configuration of the stand-alone mode of Hadoop is complete.

IV. Hadoop start

Initialization is required to start Hadoop for the first time

Change to / home/hadoop/hadoop2.8/bin directory to enter

. / hadoop namenode-format

After successful initialization, you can add a new current directory and some files in the / root/hadoop/dfs/name directory (the path is configured in the hdfs-site.xml file and created a new folder).

Starting Hadoop is mainly to start HDFS and YARN.

Change to the / home/hadoop/hadoop2.8/sbin directory

Start HDFS

Enter:

Start-dfs.sh

Login will ask if you want to connect, enter yes, and then enter the password.

Start YARN

Enter:

Start-yarn.sh

You can enter jps to see if it starts successfully.

Enter in the browser:

Http://39.108.77.250:8088/cluster

Enter:

Http://39.108.77.250:50070

Note: I forgot to take a screenshot at that time. This is a picture of a local environment built with a virtual machine.

At this point, the stand-alone configuration of Hadoop is over.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

Views: 241

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.

Share To

Internet Technology

Wechat

© 2024 shulou.com SLNews company. All rights reserved.

12
Report