Shulou(Shulou.com)06/03 Report--

As a beginner of Hadoop, it is natural to start with installation. The advantage of hadoop is distributed, so it is necessary to install a distributed system.

The overall installation steps, including Zookeeper+HDFS+Hbase, I will record my installation steps in three blog articles for the sake of brevity.

This article records the preparation of the cluster environment and the installation steps of zookeeper.

~ Environmental preparation part ~

IP deployment plan (three virtual machines prepared, operating system is Centos6.7)

Zookeeper host allocation

192.168.67.101 c6701192.168.67.102 c6702192.168.67.103 c6703

HDFS host allocation

192.168.67.101 c6701-- Namenode+datanode192.168.67.102 c6702-- datanode192.168.67.103 c6703-- datanode

Hbase host allocation

192.168.67.101 c6701-Master + region192.168.67.102 c6702-region192.168.67.103 c6703-region

1. In order to manage each service separately, I created a separate user for each service.

The users created are:

Zkhdfshbase

two。 Create ssh secret-free for each user and distribute it to all nodes, so that I will only execute commands on C6701 and install C6702 and C6703 remotely

Ssh-keygen-t rsa-f ~ / .ssh/id_rsa

Copy the key to all nodes (only c6701 does not need a password to access 02 and 03, but a password is required for reverse access)

Ssh-copy-id c6701 ssh-copy-id c6702 ssh-copy-id c6703

3. To emphasize, / etc/hosts needs to be set up

[root@c6701] # more / etc/hosts 192.168.67.101 c6701.org c6701192.168.67.102 c6702.org c6702192.168.67.103 c6703.org c6703

4. The following is executed in C6701, in which ssh is called to remotely execute some commands to verify whether ssh encryption is in effect.

Ssh c6702 "cat / proc/cpuinfo" ssh c6702 "hostname"

5. Download and install the software (intranet address)

Cd / tmp/softwarewget http://192.21.104.48/deploy/jdk-8u144-linux-x64.tar.gzwget http://192.21.104.48/deploy/zookeeper-3.4.6.tar.gzwget http://192.21.104.48/deploy/hbase-1.1.3.tar.gzwget http://192.21.104.48/deploy/hadoop-2.6.0-EDH-0u2.tar.gzwget http://192.21.104.48 / deploy/hadoop-2.7.1.tar.gz

6. Install JDK, all nodes must be installed

# tar-xzvf jdk-8u144-linux-x64.tar.gz-C / usr/local

7. Add the following information to .bash _ profile

Export JAVA_HOME=/usr/local/jdk1.8.0_144 export JRE_HOME=/usr/local/jdk1.8.0_144/jreexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATHexport PATH=$JAVA_HOME/bin:$PATHsource / etc/profile

8. Check version information

# java-versionjava version "1.8.0,144" Java (TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot (TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

~ the installation part of zookeeper ~

1. Install Zookeeper on C6701

Useradd zkecho "zk:zk" | configuration of chpasswdsu-zkmkdir zktar-zxvf / tmp/software/zookeeper-3.4.6.tar.gz-C / home/zk/zk~zoo.cfg ~ $pwd/home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg$ cat / home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg | grep-v'^ # 'tickTime=2000initLimit=10syncLimit=5dataDir=/data/zookeeper/datadataLogDir=/data/zookeeper/logclientPort=2181autopurge.snapRetainCount=3autopurge.purgeInterval=6server.1=c6701:2888:3888server.2=c6702:2888:3888server.3=c6703:2888:3888~~

two。 Create data and log folders according to zoo.cfg

# mkdir-p / data/zookeeper/data#mkdir-p / data/zookeeper/log#chown-R zk:zk / data/zookeeper#chown-R zk:zk / data/zookeeper/data#chown-R zk:zk / data/zookeeper/log

3. In the directory of zookeeper, create the above two folders. Go to the zkdata folder, create the file myid, and fill in 1. The 1 written here is the 1 in the server.1 in the zoo.cfg text. When we have configured all the files, we copy the yarn directory in hadoop1 to other machines, we are modifying the corresponding myid text in each machine, and the myid in hadoop2 is written to 2. The rest of the nodes are configured as above, and the corresponding numbers are written accordingly. The Zkdatalog folder is to specify the corresponding path for zookeeper to generate logs.

# su-zk-c "echo 1 > / data/zookeeper/data/myid"

4. Add the environment variable. In the / etc/profile directory, add ": $ZOOKEEPER_HOME/bin" to the ZOOKEEPER_HOME/bin after the original PATH

For environment variables to modify the profile file in the / etc directory, you can also add environment variables to the .bashrc directory in the root directory. What's the difference between the two: .bashrc is an environment variable for users of the current directory, and profile files are directories that are open to all users. When the system loads the file, first look for the corresponding road strength from the profile, if not, the corresponding environment variable path will be found in the .bashrc file. We all know a little about the two.

Then source / etc/profile

5. Install zookeeper for c6702

# ssh c6702 "useradd zk" # ssh c6702 "echo" zk:zk "| chpasswd"

Secret-free ssh for zk users

# ssh-copy-id zk@c6702

Copy softwar

# scp-r / tmp/software/hadoop-* root@c6702:/tmp/software#ssh c6702 "chmod 777 / tmp/software/*"

Create a directory and extract the software

# ssh zk@c6702 "mkdir zk" # ssh zk@c6702 "tar-zxvf / tmp/software/zookeeper-3.4.6.tar.gz-C / home/zk/zk" # ssh zk@c6702 "ls-al zk" # ssh zk@c6702 "ls-al zk/zookeeper*" # ssh zk@c6702 "rm / home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg" # scp-r / home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg Zk@c6702:/home/zk/zk/zookeeper-3.4.6/conf/.#ssh zk@c6702 "cat / home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg | grep-v'^ # 'create a directory based on zoo.cfg # ssh c6702" mkdir-p / data/zookeeper/data "# ssh c6702" chown-R zk:zk / data/zookeeper "# ssh c6702" chown-R zk:zk / data/zookeeper/data "# ssh c6702" mkdir-p / Data/zookeeper/log "# ssh c6702" chown-R zk:zk / data/zookeeper/log "create file myid Fill in 2ssh zk@c6702 "echo 2 > / data/zookeeper/data/myid"

6. Install zookeeper for c6703

# ssh c6703 "useradd zk" # ssh c6703 "echo" zk:zk "| chpasswd"

Secret-free ssh for zk users

Ssh-copy-id zk@c6703

Copy softwar

# scp-r / tmp/software/hadoop-* root@c6703:/tmp/software#ssh c6703 "chmod 777 / tmp/software/*"

Create a directory and extract the software

# ssh zk@c6703 "mkdir zk" # ssh zk@c6703 "tar-zxvf / tmp/software/zookeeper-3.4.6.tar.gz-C / home/zk/zk" # ssh zk@c6703 "ls-al zk" # ssh zk@c6703 "ls-al zk/zookeeper*" # ssh zk@c6703 "rm / home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg" # scp-r / home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg Zk@c6703:/home/zk/zk/zookeeper-3.4.6/conf/.#ssh zk@c6703 "cat / home/zk/zk/zookeeper-3.4.6/conf/zoo.cfg | grep-v'^ #'"

Create a directory according to zoo.cfg

# ssh c6703 "mkdir-p / data/zookeeper/data" # ssh c6703 "chown-R zk:zk / data/zookeeper" # ssh c6703 "chown-R zk:zk / data/zookeeper/data" # ssh c6703 "mkdir-p / data/zookeeper/log" # ssh c6703 "chown-R zk:zk / data/zookeeper/log" create file myid and fill in 3ssh zk@c6703 "echo 3 > / data/zookeeper/data/myid"

7. Start ZK

/ home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh start/home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh status

Remote startup command

Ssh zk@c6702 "/ home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ start" ssh zk@c6703 "/ home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ start" ssh zk@c6702 "/ home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ status" ssh zk@c6703 "/ home/zk/zk/zookeeper-3.4.6/bin/zkServer.sh\ status"

8. Not the state before all three ZK are started

[vagrant@c7003 bin] $. / zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: / home/vagrant/zookeeper-3.4.10/bin/../conf/zoo.cfgError contacting service. It is probably not running.

9. You need to start all three ZK before you can elect a leader

$. / zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: / home/vagrant/zookeeper-3.4.10/bin/../conf/zoo.cfgMode: follower [vagrant@c7002 bin] $. / zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: / home/vagrant/zookeeper-3.4.10/bin/../conf/zoo.cfgMode: leader

= 2018.1.13 update =

Client connection mode

The client connects to the server in a random order and cannot specify or which zookeeper server to read first.

. / zkCli.sh-server c6701 2181 Magazine c6702 purl 2181

The output is as follows:

First connection, connected to c6701

[zookeeper@c6702 bin] $/ usr/local/hadoop/zookeeper-3.4.6/bin/zkCli.sh-server c6701 server 2181 Magazine c6702 server 2181 Paradise 2181 Paradise 2181Connecting c6701Participating c6701Partil 2181LING c6703purl 2181.2018-01-11 21L 07purl 30797 [myid:]-INFO [main:ZooKeeper@438]-Initiating client connection, connectString=c6701:2181,c6702:2181 C6703 INFO 2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@4b4bc1eWelcome to ZooKeeperties 2018-01-11 21 INFO 07V 30830 [myid:]-INFO [main-SendThread (c6701.python279.org:2181): ClientCnxn$SendThread@975]-Opening socket connection to server c6701.python279.org/192.168.67.101:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled2018-01-11 21 myid:]-INFO [main-SendThread (c6701.python279.org:2181): ClientCnxn$SendThread@852]-Socket connection established to c6701.python279.org/192.168.67.101:2181, initiating session2018-01-11 21 V 07V 30916 [myid:]-INFO [main-SendThread (c6701.python279.org:2181): ClientCnxn$SendThread@1235]-Session establishment complete on server c6701.python279.org/192.168.67.101:2181 Sessionid = 0x160e70285f70001, negotiated timeout = 30000WATCHER::WatchedEvent state:SyncConnected type:None path:null

Second connection, connected to c6702

Connecting to c6701 INFO 2181 Magnum c6702 INFO 2181 Magazine c6703 Swiss 2181.2018-01-11 21 21 INFO 18442 [myid:]-main:ZooKeeper@438-Initiating client connection, connectString=c6701:2181,c6702:2181 C6703 INFO 2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@4b4bc1eWelcome to ZooKeeperpendent 2018-01-11 21 Jiang 10 main-SendThread 18489 [myid:]-INFO [main-SendThread (c6702.python279.org:2181): ClientCnxn$SendThread@975]-Opening socket connection to server c6702.python279.org/192.168.67.102:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled2018-01-11 21 myid 10 myid 18508 [myid:]-INFO [main-SendThread (c6702.python279.org:2181): ClientCnxn$SendThread@852]-Socket connection established to c6702.python279.org/192.168.67.102:2181, initiating session2018-01-11 21 21 10 Swiss [myid:]-INFO [main-SendThread (c6702.python279.org:2181): ClientCnxn$SendThread@1235]-Session establishment complete on server c6702.python279.org/192.168.67.102:2181 Sessionid = 0x260e70284650002, negotiated timeout = 30000WATCHER::WatchedEvent state:SyncConnected type:None path:null

At this point, the basic environment has been deployed. And the zookeeper installation is complete. We will continue to install HDFS in the next article.

Series of articles, please refer to

Hadoop Cluster (1) Zookeeper Construction

Hadoop Cluster (2) HDFS Construction

Hadoop Cluster (3) Hbase Construction

Hadoop Cluster (4) Hadoop upgrade

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.