Shulou(Shulou.com)06/02 Report--

I. Environmental preparation

Operating system version:

Centos7 64 bit

Hadoop version:

Hadoop-2.7.3

Hbase version:

Hbase-1.2.4

1. Machines and roles

IP role 192.168.11.131NamenodeDFSZKFailoverControllerResourceManagerJobhistory

HMasterQuorumPeerMain192.168.11.132NamenodeDFSZKFailoverController

HMasterQuorumPeerMain192.168.11.133

DatanodeNodeManagerJournalNodeHRegionServer

QuorumPeerMain192.168.11.134

DatanodeNodeManagerJournalNodeHRegionServer

QuorumPeerMain192.168.11.135

DatanodeNodeManagerJournalNodeHRegionServer

QuorumPeerMain

Master1:

# hostnamectl set-hostname master1

Other hostnames are configured as master2, slave1, slave2, and slave3

2. All nodes turn off firewall and selinux

Description:

Before the command, # indicates the operation under the root user, and $indicates the operation under the hduser user.

View firewall status

# firewall-cmd-state

Running

Turn off the firewall

# systemctl stop firewalld.service

Firewall is configured to be disabled

# systemctl disable firewalld.service

Close selinux immediately (temporary)

# setenforce 0

Permanently close selinux

# vi / etc/sysconfig/selinux

SELINUX=enforcing-- > disabled

3. Configure yum source

All nodes are configured with yum source

# cd

# mkdir apps

Http://mirrors.163.com/centos/7/os/x86_64/Packages/wget-1.14-15.el7.x86_64.rpm

# rpm-I wget-1.14-15.el7.x86_64.rpm

# cd / etc/yum.repos.d

# wget http://mirrors.aliyun.com/repo/Centos-7.repo

# mv Centos-7.repo CentOS-Base.repo

# scp CentOS-Base.repo root@192.168.11.131:/etc/yum.repos.d/

# scp CentOS-Base.repo root@192.168.11.132:/etc/yum.repos.d/

# scp CentOS-Base.repo root@192.168.11.133:/etc/yum.repos.d/

# scp CentOS-Base.repo root@192.168.11.134:/etc/yum.repos.d/

# yum clean all

# yum makecache

# yum update

4. Configure ntp time synchronization. Master1 is the ntp server, and the others are ntp clients.

All nodes install ntp

# yum install-y ntp

Ntp server side:

First configure the current time

# date-s "2018-05-27 23:03:30"

Edit configuration file

# vi / etc/ntp.conf

Add two lines under the comment

# restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

Server 127.127.1.0

Fudge 127.127.1.0 stratum 11

Below the comment

# server 0.centos.pool.ntp.org iburst

# server 1.centos.pool.ntp.org iburst

# server 2.centos.pool.ntp.org iburst

# server 3.centos.pool.ntp.org iburst

Start the ntp service and configure it to boot automatically

# systemctl start ntpd.service

# systemctl enable ntpd.service

Ntp client (the other four are ntp clients):

# vi / etc/ntp.conf

Add two lines under the same comment

# restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

Server 192.168.11.131

Fudge 127.127.1.0 stratum 11

Add comments on four lines

# server 0.centos.pool.ntp.org iburst

# server 1.centos.pool.ntp.org iburst

# server 2.centos.pool.ntp.org iburst

# server 3.centos.pool.ntp.org iburst

# systemctl start ntpd.service

# systemctl enable ntpd.service

Manual synchronization time

# ntpdate 192.168.11.131

28 May 07:04:50 ntpdate [1714]: the NTP socket is in use, exiting

Deal with the error report.

# lsof-iPUR 123

-bash: lsof: command not found

# yum install-y lsof

# lsof-iPUR 123

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

Ntpd 1693 ntp 16u IPv4 25565 0t0 UDP *: ntp

Ntpd 1693 ntp 17u IPv6 25566 0t0 UDP *: ntp

Ntpd 1693 ntp 18u IPv4 25572 0t0 UDP localhost:ntp

Ntpd 1693 ntp 19u IPv4 25573 0t0 UDP localhost.localdomain:ntp

Ntpd 1693 ntp 20u IPv6 25574 0t0 UDP localhost:ntp

Ntpd 1693 ntp 21u IPv6 25575 0t0 UDP localhost.localdomain:ntp

To kill the ntp process, note that the ntp process id is different and should be handled according to the actual situation.

# kill-9 1693

Synchronize time again successfully

# ntpdate 192.168.11.131

27 May 23:06:14 ntpdate [1728]: step time server 192.168.11.131 offset-28808.035509 sec

5. Modify the hostname and hosts file

All nodes modify hostname (permanent)

# hostnamectl set-hostname master1~slave3

Temporarily modify the hostname

# hostname master1~slave3

Master node modifies hosts file

# vi / etc/hosts

192.168.11.131 master1

192.168.11.132 master2

192.168.11.133 slave1

192.168.11.134 slave2

192.168.11.135 slave3

Overwrite hosts files to other machines

# scp / etc/hosts root@192.168.11.132~135:/etc/

All nodes create administrative users and groups

Create groups and users

# groupadd hduser

# useradd-g hduser hduser

# passwd hduser

2. Configure the hadoop environment

1. Create a directory and give it weight

Create the following folders on each machine

# mkdir / data1

# mkdir / data2

Modify permissions

# chown hduser:hduser / data1

# chown hduser:hduser / data2

# su hduser

$mkdir-p / data1/hadoop_data/hdfs/namenode

$mkdir-p / data2/hadoop_data/hdfs/namenode

$mkdir-p / data1/hadoop_data/hdfs/datanode (NameNode do not)

$mkdir-p / data2/hadoop_data/hdfs/datanode (NameNode do not)

$mkdir-p / data1/hadoop_data/pids

$mkdir-p / data2/hadoop_data/pids

$mkdir-p / data1/hadoop_data/hadoop_tmp

$mkdir-p / data2/hadoop_data/hadoop_tmp

2. No secret verification

Master1 and master2 node operation

# su-hduser

$ssh-keygen-t rsa

$cd ~ / .ssh

$cat id_rsa.pub > > authorized_keys

Master1 node operation

$ssh-copy-id-I ~ / .ssh/id_rsa.pub hduser@master2

Master2 node operation

$scp ~ / .ssh/authorized_keys hduser@master1:~/.ssh/

The slave1, slave2, and slave3 nodes create .ssh directories

# mkdir / home/hduser/.ssh

# chown hduser:hduser / home/hduser/.ssh

Master1 node operation

$scp ~ / .ssh/authorized_keys hduser@slave1:~/.ssh

$scp ~ / .ssh/authorized_keys hduser@slave2:~/.ssh

$scp ~ / .ssh/authorized_keys hduser@slave3:~/.ssh

Master1 and master2 node authentication

Verification method, respectively, in two nodes, ssh login native (hdusser users) and the other four nodes to see if there is no secret login.

If the verification fails, all machines execute the following command

$chmod 600 ~ / .ssh/authorized_keys

$chmod 700 ~ / .ssh

3. All nodes are configured with java environment

$mkdir-p / data1/usr/src

Upload the package to the / data1/usr/src directory

$cd / data1/usr/src

$tar xf jdk1.7.0_79.tar-C / data1/usr/

$vi ~ / .bashrc

Export JAVA_HOME=/data1/usr/jdk1.7.0_79

Export JRE_HOME=$JAVA_HOME/jre

Export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib/rt.jar

Export PATH=$PATH:$JAVA_HOME/bin

$source ~ / .bashrc

4. Mastar1 node configures hadoop

Download hadoop-2.7.3.tar.gz and upload to / data1/usr/src

Http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

$cd / data1/usr/src

$tar-zxf hadoop-2.7.3.tar.gz-C / data1/usr/

Edit hadoop-env.sh

$vi / data1/usr/hadoop-2.7.3/etc/hadoop/hadoop-env.sh

Export JAVA_HOME=/data1/usr/jdk1.7.0_79

Export HADOOP_PID_DIR=/data1/hadoop_data/pids

Export HADOOP_PID_DIR=/data2/hadoop_data/pids

Export HADOOP_MAPRED_PID_DIR=/data1/hadoop_data/pids

Edit mapred-env.sh

$vi / data1/usr/hadoop-2.7.3/etc/hadoop/mapred-env.sh

Export HADOOP_MAPRED_PID_DIR=/data2/hadoop_data/pids

Edit yarn-env.sh

$vi / data1/usr/hadoop-2.7.3/etc/hadoop/yarn-env.sh

Export YARN_PID_DIR=/data2/hadoop_data/pids

Edit core-site.xml

$vi / data1/usr/hadoop-2.7.3/etc/hadoop/core-site.xml

Fs.defaultFS

Hdfs://masters

Hadoop.tmp.dir

/ data2/hadoop_data/hadoop_tmp

Ha.zookeeper.quorum

Master1:2181,master2:2181,slave1:2181,slave2:2181,slave3:2181

Edit hdfs-site.xml

$vi / data1/usr/hadoop-2.7.3/etc/hadoop/hdfs-site.xml

Dfs.nameservices

Masters

Dfs.ha.namenodes.masters

Master1,master2

Dfs.namenode.rpc-address.masters.master1

Master1:9000

Dfs.namenode.http-address.masters.master1

Master1:50070

Dfs.namenode.rpc-address.masters.master2

Master2:9000

Dfs.namenode.http-address.masters.master2

Master2:50070

Dfs.namenode.name.dir

File:///data2/hadoop_data/hdfs/namenode

Dfs.datanode.data.dir

File:///data1/hadoop_data/hdfs/datanode,data2/hadoop_data/hdfs/datanode

Dfs.namenode.shared.edits.dir

Qjournal://slave1:8485;slave2:8485;slave3:8485/masters

Dfs.journalnode.edits.dir

/ data2/hadoop_data/journal

Dfs.ha.automatic-failover.enabled

True

Dfs.client.failover.proxy.provider.masters

Org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

Dfs.ha.fencing.methods

Sshfence

Shell (/ bin/true)

Dfs.ha.fencing.ssh.private-key-files

/ home/hduser/.ssh/id_rsa

Dfs.ha.fencing.ssh.connect-timeout

30000

Dfs.datanode.max.xcievers

8192

Dfs.qjournal.write-txns.timeout.ms

60000

Edit yarn-site.xml

$vi / data1/usr/hadoop-2.7.3/etc/hadoop/yarn-site.xml

Yarn.resourcemanager.ha.enabled

True

Yarn.resourcemanager.cluster-id

RM_HA_ID

Yarn.resourcemanager.ha.rm-ids

Rm1,rm2

Yarn.resourcemanager.hostname.rm1

Master1

Yarn.resourcemanager.hostname.rm2

Master2

Yarn.resourcemanager.recovery.enabled

True

Yarn.resourcemanager.store.class

Org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore

Yarn.resourcemanager.zk-address

Master1:2181,master2:2181,slave1:2181,slave2:2181,slave3:2181

Yarn.nodemanager.aux-services

Mapreduce_shuffle

Yarn.nodemanager.aux-services.mapreduce.shuffle.class

Org.apache.hadoop.mapred.ShuffleHandler

Edit mapred-site.xml

$cp / data1/usr/hadoop-2.7.3/etc/hadoop/mapred-site.xml.template / data1/usr/hadoop-2.7.3/etc/hadoop/mapred-site.xml

$vi / data1/usr/hadoop-2.7.3/etc/hadoop/mapred-site.xml

Mapreduce.framework.name

Yarn

Edit slaves

$vi / data1/usr/hadoop-2.7.3/etc/hadoop/slaves

Slave1

Slave2

Slave3

Synchronize hadoop configuration to other nodes

$for ip in `seq 2 5`; do scp-rpq / data1/usr/hadoop-2.7.3 192.168.11.13$ ip:/data1/usr;done

5. Zookeeper configuration of each node

Http://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

Upload the package to the / data1/usr/src directory

Create a directory

$mkdir-p / home/hduser/storage/zookeeper

$cd / data1/usr/src

$tar-zxf zookeeper-3.4.6.tar.gz-C / data1/usr

Edit zoo.cfg

$cp / data1/usr/zookeeper-3.4.6/conf/zoo_sample.cfg / data1/usr/zookeeper-3.4.6/conf/zoo.cfg

$vi / data1/usr/zookeeper-3.4.6/conf/zoo.cfg

DataDir=/home/hduser/storage/zookeeper

Server.1=master1:2888:3888

Server.2=master2:2888:3888

Server.3=slave1:2888:3888

Server.4=slave2:2888:3888

Server.5=slave3:2888:3888

Each node of master1-slave3 operates sequentially.

$echo "1" > / home/hduser/storage/zookeeper/myid

$echo "2" > / home/hduser/storage/zookeeper/myid

$echo "3" > / home/hduser/storage/zookeeper/myid

$echo "4" > / home/hduser/storage/zookeeper/myid

$echo "5" > / home/hduser/storage/zookeeper/myid

Each node starts the ZK service

$cd / data1/usr/zookeeper-3.4.6/bin

$. / zkServer.sh start

6. Slave1, slave2 and slave3 start journalnode

$cd / data1/usr/hadoop-2.7.3/sbin

$. / hadoop-daemon.sh start journalnode

Confirm the startup result with jps

7. Format zookeeper nodes on master1 (for the first time)

$cd / data1/usr/hadoop-2.7.3

. / bin/hdfs zkfc-formatZK

8. Execute the command on master1 to format the file system

. / bin/hadoop namenode-format

9. Start the namenode and zkfc services on master1

. / sbin/hadoop-daemon.sh start namenode

. / sbin/hadoop-daemon.sh start zkfc

10. Data synchronization needs to be performed on the master2 (standby node)

. / bin/hdfs namenode-bootstrapStandby

Copy files to master2 on master1

Scp-r / data2/hadoop_data/hdfs/namenode hduser@master2:/data2/hadoop_data/hdfs/

11. Start the namenode and zkfc services on master2

. / sbin/hadoop-daemon.sh start namenode

. / sbin/hadoop-daemon.sh start zkfc

12. Set master1 to active

. / bin/hdfs haadmin-transitionToActive master1

. / bin/hdfs haadmin-getServiceState master1

13. Start datanode on master1

. / sbin/hadoop-daemons.sh start datanode

14. Start HDFS (after the second time)

Execute the command on master1:

. / sbin/start-dfs.sh

15. Start YARN

Execute the command on master1:

. / sbin/start-yarn.sh

16. Start Jobhistory

. / sbin/mr-jobhistory-daemon.sh start historyserver

17. Verification

Verify namenode

Http://master1:50070

Overview 'master1:9000' (active)

Http://master2:50070

Overview 'master2:9000' (standby)

18. Verify the file system

Upload files

. / bin/hadoop fs-put / data1/usr/hadoop-2.7.3/etc/hadoop / test/hadoop

. / bin/hadoop fs-ls / test

19. Backup verification of namenode

Kill master1,master2 and become active.

20. Verify yarn

. / bin/hadoop jar / data1/usr/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount / test/hadoop / test/out

Third, install HBASE

Download hbase-1.2.4-bin.tar.gz and extract it

$cd / data1/usr/src

$tar-zxvf hbase-1.2.4-bin.tar.gz-C / data1/usr/

3. Create a directory

$mkdir-p / data1/hadoop_data/hbase_tmp

$mkdir-p / data2/hadoop_data/hbase_tmp

2. Configure the hbase environment of master1

Configure hbase-env.sh

$vi / data1/usr/hbase-1.2.4/conf/hbase-env.sh

Export JAVA_HOME=/data1/usr/jdk1.7.0_79

Export HBASE_PID_DIR=/data2/hadoop_data/pids

Export HBASE_MANAGES_ZK=false

Export HADOOP_HOME=/data1/usr/hadoop-2.7.3

3. Configure hbase-site.xml

$vi / data1/usr/hbase-1.2.4/conf/hbase-site.xml

Hbase.rootdir

Hdfs://masters/hbase

Hbase.cluster.distributed

True

Hbase.master.port

60000

Hbase.tmp.dir

/ data2/hadoop_data/hbase_tmp

Hbase.zookeeper.quorum

Master1,master2,slave1,slave2,slave3

4. Configure regionservers

$vi / data1/usr/hbase-1.2.4/conf/regionservers

Slave1

Slave2

Slave3

5. Configure backup-masters

$vi / data1/usr/hbase-1.2.4/conf/backup-masters

Master2

6. Remove the jar package of unnecessary log4j in HBase

Cd ${HBASE_HOME} / lib

Mv slf4j-log4j12-1.7.5.jar slf4j-log4j12-1.7.5.jar.bak

7. Transfer the hbase environment of master1 to other nodes

$for ip in `seq 2 5`; do scp-rpq / data1/usr/hbase-1.2.4 192.168.11.13$ ip:/data1/usr;done

8. Start Hbase on master1 (hadoop cluster has been started)

$cd / data1/usr/hbase-1.2.4/bin

$. / start-hbase.sh

9. Verification

Check to see if hbase is created successfully on the HDFS file system

$/ data1/usr/hadoop-2.7.3/bin/hadoop fs-ls /

Execution: bin/hbase shell can enter the Hbase management interface,

Enter status to view status

Create a tabl

Create 'test',' cf'

Display table information

List 'test'

Insert data into the table

Put 'test',' row1', 'cf:a',' value1'

Put 'test',' row2', 'cf:b',' value2'

Put 'test',' row3', 'cf:c',' value3'

Query table

Scan 'test'

Fetch a row of data

Get 'test',' row1'

Failure table

Disable 'test'

Delete tabl

Drop 'test'

Enter http://master1:16010 in the browser to open the Hbase management interface

Http://192.168.11.131/master-status

10. Start thrift2

Hbase-daemons.sh start thrift2

Each node uses jps to confirm whether the service is started. Generally, the reason for startup failure is misconfiguration.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.