ZooKeeper for Hadoop deployment (2)

I. brief introduction of Zookeeper function

ZooKeeper is an open source distributed coordination service created by Yahoo and is an open source implementation of Google Chubby. Distributed applications can implement functions such as data publish / subscribe, load balancing, naming service, distributed coordination / notification, cluster management, Master election, configuration maintenance, name service, distributed synchronization, distributed locking and distributed queue based on ZooKeeper.

2. Basic concepts of ZooKeeper 1. Cluster role

A ZooKeeper cluster will have only one Leader at a time, and the others are Follower or Observer.

ZooKeeper configuration is simple, the configuration file (zoo.cfg) of each node is the same, only the myid file is different. The value of myid must be the {numeric} part of server. {Numeric} in zoo.cfg.

By default, ZooKeeper has only two roles: Leader and Follower, and there is no Observer role. To use Observer mode, add: peerType=observer to the configuration file of any node that wants to become Observer and in the configuration files of all server, the line configuration of server configured as observer mode appends: observer

3. Installation of ZooKeeper 1. Download and install (install on datanode node)

Download the installation package

Wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz

Extract the installation package

Tar xf zookeeper-3.4.10.tar.gzmv zookeeper-3.4.10 / usr/local/zookeepermkdir-p / home/zookeeper

Zk01 node (datanode1) configure id

Echo "1" > / home/zookeeper/myid

Zk02 node (datanode2) configure id

Echo "2" > / home/zookeeper/myid

Zk03 node (datanode3) configure id

Echo "3" > / home/zookeeper/myid2, configure ZooKeeper environment variables

Edit the file / etc/profile.d/zookeeper.sh.

# ZOOKEEPER ENVexport ZOOKEEPER_HOME=/usr/local/zookeeperexport PATH=$PATH:$ZOOKEEPER_HOME/bin

Make the ZK environment variable effective.

Source / etc/profile.d/zookeeper.sh3, configure zoo.cfg

Edit the file / usr/local/zookeeper/conf/zoo.cfg.

# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial # synchronization phase can takeinitLimit=10# The number of ticks that can pass between # sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use / tmp for storage / tmp here is just # example sakes.dataDir=/home/zookeeper# the port at which the clients will connectclientPort=2181# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the # administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to "0" to disable autopurge feature#autopurge. PurgeInterval=1server.1=zk01:2888:3888server.2=zk02:2888:3888server.3=zk03:2888:3888

The more important configurations are as follows:

DataDir=/home/zookeeper

Server.1=zk01:2888:3888

Server.2=zk02:2888:3888

Server.3=zk03:2888:3888

4. Start the ZK service of each node [root@datanode01 ~] # zkServer.sh startZooKeeper JMX enabled by defaultUsing config: / usr/local/zookeeper/bin/../conf/zoo.cfgStarting zookeeper. STARTED [root@datanode02 ~] # zkServer.sh startZooKeeper JMX enabled by defaultUsing config: / usr/local/zookeeper/bin/../conf/zoo.cfgStarting zookeeper. STARTED [root@datanode03 ~] # zkServer.sh startZooKeeper JMX enabled by defaultUsing config: / usr/local/zookeeper/bin/../conf/zoo.cfgStarting zookeeper. STARTED5, check the startup status and roles of each node [root@datanode01 ~] # zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: / usr/local/zookeeper/bin/../conf/zoo.cfgMode: follower [root@datanode02 ~] # zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: / usr/local/zookeeper/bin/../conf/zoo.cfgMode: leader [root@datanode03 ~] # zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: / usr/local/zookeeper/bin/../conf/zoo.cfgMode: follower

Official document: https://zookeeper.apache.org/doc/current/zookeeperStarted.html