Shulou(Shulou.com)06/01 Report--

This article mainly introduces the relevant knowledge of "how to build a Zookeeper cluster". The editor shows you the operation process through an actual case. The operation method is simple, fast and practical. I hope this article "how to build a Zookeeper cluster" can help you solve the problem.

A brief introduction to the principle of Zookeeper

ZooKeeper is an open source distributed application coordination service, which contains a simple set of primitives based on which distributed applications can implement synchronization services, configuration maintenance and naming services.

Design purpose of Zookeeper

Ultimate consistency: no matter which Server client connects to, it shows it the same view. Reliability: with simple, robust, good performance, if message m is received by one server, then message m will be received by all servers. Real-time: Zookeeper ensures that the client will get the update information of the server or the information of server failure within a time interval. However, due to network delay and other reasons, Zookeeper can not guarantee that two clients can get the newly updated data at the same time. If the latest data is needed, the sync () interface should be called before reading the data. Wait irrelevant (wait-free): slow or invalid client must not interfere with fast client requests so that each client can wait effectively. Atomicity: updates can only succeed or fail, with no intermediate state. Ordering: including global order and partial order: global order means that if message an is published before message b on a server, message a will be published before message b on all Server; partial order means that if a message b is released by the same sender after message a, a will be before b.

How Zookeeper works

1. In the zookeeper cluster, each node has the following three roles and four states:

Role: leader,follower,observer status: leading,following,observing,looking

The core of Zookeeper is atomic broadcasting, which ensures synchronization between Server. The protocol that implements this mechanism is called the Zab protocol (ZooKeeper Atomic Broadcast protocol). There are two modes of Zab protocol, which are recovery mode (Recovery primary) and broadcast mode (Broadcast synchronization). When the service starts or after the leader crashes, the Zab enters the recovery mode, and when the leader is elected and most of the Server finishes synchronizing with the leader, the recovery mode ends. State synchronization ensures that leader and Server have the same system state.

In order to ensure the order consistency of transactions, zookeeper uses an incremental transaction id number (zxid) to identify transactions. All proposals (proposal) are made with zxid when they are made. In the implementation, zxid is a 64-bit number, and its high 32-bit epoch is used to identify whether the leader relationship has changed. Each time a leader is selected, it will have a new epoch that identifies the current period of leader rule. The lower 32 bits are used to increment the count.

Each Server has four states during its operation:

LOOKING: currently Server doesn't know who leader is and is searching for it.

LEADING: the current Server is the elected leader.

FOLLOWING:leader has been elected and the current Server is synchronized with it.

The behavior of OBSERVING:observer is exactly the same as that of follower in most cases, but they do not participate in elections and votes, but only accept (observing) the results of elections and ballots.

Zookeeper cluster node

The more Zookeeper nodes are deployed, the higher the reliability of the service. It is recommended to deploy an odd number of nodes, because it takes more than half of the outages of zookeeper clusters to bring down the entire cluster. You need to give each zookeeper about 1G of memory, and if possible, it's best to have a separate disk, because it ensures that zookeeper is high-performance. If your cluster is heavily loaded, don't run zookeeper and RegionServer on the same machine, just like DataNodes and TaskTrackers.

Experimental environment

Hostname system IP address linux-node1CentOS release 6.8192.168.1.148linux-node2CentOS release 6.8192.168.1.149linux-node2CentOS release 6.8192.168.1.150 II, Zookeeper installation

Zookeeper requires java environment and jdk needs to be installed. Note: zookeeper and jdk need to be installed on each server. It is recommended to download the required installation package locally and upload it to the server. The download speed on the server is too slow.

2.1The installation of JDK

Rpm-ivh jdk-8u101-linux-x64.rpm

2.2, Zookeeper installation

Wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz-P / usr/local/src/tar zxvf zookeeper-3.4.8.tar.gz-C / optcd / opt & & mv zookeeper-3.4.8 zookeepercd zookeepercp conf/zoo_sample.cfg conf/zoo.cfg

# add zookeeper to the environment variable

Echo-e "# append zk_env\ nexport PATH=$PATH:/opt/zookeeper/bin" > > / etc/profile III. Zookeeper cluster configuration

Note: when building a zookeeper cluster, be sure to stop the zookeeper nodes that have been started.

3.1.Modification of Zookeeper configuration file

# the modified configuration file zoo.cfg is as follows:

Egrep-v "^ # | ^ $" zoo.cfgtickTime=2000initLimit=10syncLimit=5dataLogDir=/opt/zookeeper/logsdataDir=/opt/zookeeper/dataclientPort=2181autopurge.snapRetainCount=500autopurge.purgeInterval=24server.1= 192.168.1.148MAV 2888server.2 = 192.168.1.149MUR 2888server.3 = 192.168.1.150MAV 2888server.3 = 192.168.1.150

# create related directories, all three nodes need

Mkdir-p / opt/zookeeper/ {logs,data}

# after the installation of the remaining zookeeper nodes is complete, synchronize the configuration file zoo.cfg.

3.2. Description of configuration parameters

TickTime this time is used as the interval between zookeeper servers or between clients and servers to maintain a heartbeat, which means that a heartbeat is sent for each tickTime time.

InitLimit is used to configure the zookeeper accept client (the client here is not the client that the user connects to the zookeeper server, but the follower server in the zookeeper server cluster that connects to the leader) the maximum number of heartbeat intervals that can be tolerated when initializing the connection.

When the zookeeper server has not received a return message from the client after the length of more than 10 heartbeats (that is, tickTime), the client connection failed. The total length of time is 10 "2000" 20 seconds.

SyncLimit this configuration item identifies the length of time for sending messages, requests and replies between leader and follower. The maximum length of time cannot exceed the number of tickTime. The total length of time is 5, 000, 000, 10 seconds.

DataDir, as its name implies, is the directory where zookeeper saves data. By default, zookeeper also saves log files for writing data in this directory.

ClientPort port is the port through which the client connects to the Zookeeper server. Zookeeper listens on this port to accept client access requests.

An in server.A=B:C:D is a number, indicating which server this is, B is the IP address of this server, the first port of C is used for the exchange of information among cluster members, indicating the port on which this server exchanges information with the leader server in the cluster, and D is the port used specifically to elect leader when the leader is hung up.

3.3.Create ServerID logo

In addition to modifying the zoo.cfg configuration file, a myid file needs to be configured in zookeeper cluster mode, which needs to be placed in the dataDir directory.

One of the data in this file is the value of A (which is the An in the server.A=B:C:D in the zoo.cfg file), and the myid file is created in the dataDir path configured in the zoo.cfg file.

# create a myid file on the 192.168.1.148 server and set the value to 1, which is consistent with the server.1 in the zoo.cfg file, as follows

Echo "1" > / opt/zookeeper/data/myid

# create a myid file on the 192.168.1.149 server and set the value to 1, which is consistent with the server.2 in the zoo.cfg file, as follows

Echo "2" > / opt/zookeeper/data/myid

# create a myid file on the 192.168.1.150 server and set the value to 1, which is consistent with the server.3 in the zoo.cfg file, as follows

Echo "3" > / opt/zookeeper/data/myid

At this point, the relevant configuration has been completed.

4. Zookeeper cluster view

1. Start the zookeeper node on each server:

# linux-node1 、 linux-node2 、 linux-node3

/ opt/zookeeper/bin/zkServer.sh start

Note: error troubleshooting

The possible reason why the Zookeeper node cannot be started is that the zoo.cfg configuration file is incorrect and the iptables is not closed.

2. Check the status of each node after startup

# linux-node1

# linux-node2

# linux-node3

# as can be seen from the above, the status of linux-node1,linux-node3 server zookeeper is follow mode, and that of linux-node2 server zookeeper is leader mode.

* * 5. Zookeeper cluster connection

**

After the Zookeeper cluster is built, you can connect to the zookeeper cluster through client script. For the client, the zookeeper cluster is a whole, and connecting to the zookeeper cluster actually feels like enjoying the services of the entire cluster. # in the linux-node1 test, the above figure shows that the entire zookeeper cluster has been built and the test has been completed.

This is the end of the introduction to "how to build Zookeeper clusters". Thank you for reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.