Shulou(Shulou.com)06/03 Report--

The software required for this cluster:

VMwareworkstation64_12.5.7.0

Xmanager Enterprise 4 Build 0182

CentOS-7-x86_64-DVD-1708

Hadoop-2.8.2.tar

Jdk-8u151-linux-x64.tar

Three mainframes

Master 192.168.1.140

Slave1 192.168.1.141

Slave2 192.168.1.142

1. Install a virtual machine

(1) install a virtual machine

(2) Clone a virtual machine

two。 Modify hostname

Hostnamectl set-hostname master (slave1,slave2)

3. Modify IP

Vi / etc/sysconfig/network-scripts/ifcfg-ens33

ONBOOT=yes

IPADDR=192.168.1.140

NETMASK=255.255.255.0

GATEWAY=192.168.1.1

DNS=192.168.1.254

4. Modify / etc/hosts file

Add

192.168.1.140 master

192.168.1.141 slave1

192.168.1.142 slave2

5. Turn off the firewall

Systemctl stop firewalld

Forbid the firewall

Systemctl disable firewalld

6. Modify SELinux

Vi / etc/sysconfig/selinux

SELinux=disabled

7. Password-free login

Ssh-keygen then kept on entering the car.

Redirect id_rsa.pub to authorized_keys on master

Cat id_rsa.pub > > authorized_keys

Distribute authorized_keys to the / root/.ssh directory of slaver1 and slaver2

IP address of scp authorized_keys slaver1/slaver2 / root/.ssh/

8. Uninstall jdk

Rpm-qa | grep java

Delete all java

Rpm-e java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64-- nodeps

9.yum source configuration

Cd / run/media/root/CentOS\ 7\ x86_64/Packages/

Cp * / yum

Cd / yum

Rpm- ivh deltarpm-3.6-3.el7.x86_64.rpm

Rpm- ivh python-deltarpm-3.6-3.el7.x86_64.rpm

Rpm-ivh createrepo-0.9.9-28.el7.noarch.rpm

Createrepo.

Cd / etc/yum.repos.d/

Rm-rf *

Vi yum.repo

[local]

Name=yum.repo

Baseurl= file:///yum

Gpgcheck=0

Enabled=1

Yum clean all

Yum repolist

10. Configure the ftp service

Yum-y install ftp* vsftpd*

Baseurl= ftp://192.168.1.140/pub for slave yum source

11. Install the Yum package

Yum-y install libc*

Yum-y install openssh*

Yum-y install man*

Yum-y install compat-libstdc++-33*

Yum-y install libaio-0.*

Yum-y install libaio-devel*

Yum-y install sysstat-9.*

Yum-y install glibc-2.*

Yum-y install glibc-devel-2.* glibc-headers-2.*

Yum-y install ksh-2*

Yum-y install libgcc-4.*

Yum-y install libstdc++-4.*

Yum-y install libstdc++-4.*.i686*

Yum-y install libstdc++-devel-4.*

Yum-y install gcc-4.*x86_64*

Yum-y install gcc-c++-4.*x86_64*

Yum-y install elfutils-libelf-0*x86_64* elfutils-libelf-devel-0*x86_64*

Yum-y install elfutils-libelf-0*i686* elfutils-libelf-devel-0*i686*

Yum-y install libtool-ltdl*i686*

Yum-y install ncurses*i686*

Yum-y install ncurses*

Yum-y install readline*

Yum-y install unixODBC*

Yum-y install zlib

Yum-y install zlib*

Yum-y install openssl*

Yum-y install patch*

Yum-y install git*

Yum-y install lzo-devel* zlib-devel* gcc* autoconf* automake* libtool*

Yum-y install lzop*

Yum-y install lrzsz*

Yum-y install lzo-devel* zlib-devel* gcc* autoconf* automake* libtool*

Yum-y install nc*

Yum-y install glibc*

Yum-y install gzip*

Yum-y install zlib*

Yum-y install gcc*

Yum-y install gcc-c++*

Yum-y install make*

Yum-y install protobuf*

Yum-y install protoc*

Yum-y install cmake*

Yum-y install openssl-devel*

Yum-y install ncurses-devel*

Yum-y install unzip*

Yum-y install telnet*

Yum-y install telnet-server*

Yum-y install wget*

Yum-y install svn*

Yum-y install ntpdate*

Turn off unnecessary services

Chkconfig autofs off

Chkconfig acpid off

Chkconfig sendmail off

Chkconfig cups-config-daemon off

Chkconfig cpus off

Chkconfig xfs off

Chkconfig lm_sensors off

Chkconfig gpm off

Chkconfig openibd off

Chkconfig pcmcia off

Chkconfig cpuspeed off

Chkconfig nfslock off

Chkconfig iptables off

Chkconfig ip6tables off

Chkconfig rpcidmapd off

Chkconfig apmd off

Chkconfig sendmail off

Chkconfig arptables_jf off

Chkconfig microcode_ctl off

Chkconfig rpcgssd off

twelve。 Install Java

Upload the jdk-8u151-linux-x64.tar.gz installation package to / usr/local

Decompression tar xzvf jdk completion

Rename mv jdk1.8.0_151/ java

Modify / etc/profile file

Export JAVA_HOME=/usr/local/java

Export JRE_HOME=/usr/java/jre

Export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib

Export PATH=$PATH:$JAVA_HOME/bin

Generate environment variables

Source / etc/profile

The scp command distributes the entire folder to laver2 on slave nodes slaver1, slaver2

Scp-r / etc/profile root@slaver1 (slave2): / etc/profile

13.hadoop installation

Upload Hadoop package to / usr/local/

Decompress tar xzvf hadoop-2.8.2.tar.gz

Rename mv hadoop-2.8.2 hadoop

Modify / etc/profile file

Export HADOOP_HOME=/usr/local/hadoop

# export HADOOP_OPTS= "- Djava.library.path=$HADOOP_PREFIX/lib:$HADOOP_PREFIX/lib/native"

Export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native

Export HADOOP_COMMON_LIB_NATIVE_DIR=/usr/local/hadoop/lib/native

Export HADOOP_OPTS= "- Djava.library.path=/usr/local/hadoop/lib"

# export HADOOP_ROOT_LOGGER=DEBUG,console

Export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Generate environment variables

Source / etc/profile

Modify the configuration file

Cd / usr/local/hadoop/etc/hadoop

(1) vi hadoop-env.sh

Modify content export JAVA_HOME=/usr/local/java

(2) vi core-site.xml

Fs.defaultFS

Hdfs://192.168.1.140:9000

Io.file.buffer.size

131072

Hadoop.tmp.dir

File:/usr/local/hadoop/tmp

Abase for other temporary directories.

Hadoop.proxyuser.root.hosts

*

Hadoop.proxyuser.root.groups

*

(3) modify yarn-site.xml file

Vi yarn-site.xml

Yarn.nodemanager.aux-services

Mapreduce_shuffle

Yarn.nodemanager.aux-services.mapreduce.shuffle.class

Org.apache.hadoop.mapred.ShuffleHandler

Yarn.resourcemanager.address

192.168.1.140:8032

Yarn.resourcemanager.scheduler.address

192.168.1.140:8030

Yarn.resourcemanager.resource-tracker.address

192.168.1.140:8035

Yarn.resourcemanager.admin.address

192.168.1.140:8033

Yarn.resourcemanager.webapp.address

192.168.1.140:8088

(4) modify hdfs-site.xml file

Dfs.namenode.name.dir

File:/usr/local/hadoop/hdfs/name

Dfs.datanode.data.dir

File:/usr/local/hadoop/hdfs/data

Dfs.replication

three

Dfs.namenode.secondary.http-address

192.168.1.140:9001

Dfs.webhdfs.enabled

True

(5) modify mapred-site.xml

Mapreduce.framework.name

Yarn

(6) modify slaves file

192.168.1.140

192.168.1.141

192.168.1.142

14. Send the file configured by master to slave1,slave2 and generate environment variables

15. Format hadoop

Hadoop namenode-format

16. Start the cluster

Start-all.sh

Master:

[root@master ~] # jps

4705 SecondaryNameNode

8712 Jps

4299 NameNode

4891 ResourceManager

5035 NodeManager

4447 DataNode

Slave1:

[root@slave1 ~] # jps

2549 DataNode

2729 NodeManager

3258 Jps

Slave2:

[root@slave2 ~] # jps

3056 Jps

2596 DataNode

2938 NodeManager

17. Enter the master address +: 8088 in the browser (MR management interface)

Enter the master address +: 50070 in the browser (HDFS management interface)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.