Shulou(Shulou.com)06/02 Report--

What are the functions of Zookeeper, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

What is 1.ZooKeeper?

ZooKeeper is a distributed, open source distributed application coordination service. It is an open source implementation of Google's Chubby. It is the manager of the cluster, monitoring the status of each node in the cluster and carrying out the next reasonable operation according to the feedback submitted by the node. Finally, the user will be provided with an easy-to-use interface and a system with efficient performance and stable function.

What does 2.ZooKeeper provide?

1) File system

2) Notification mechanism

3.Zookeeper file system

Every subdirectory entry such as NameService is called znode. Like the file system, we are free to add and delete znode, and add and delete sub-znode under a znode. The only difference is that znode can store data.

There are four types of znode:

1. PERSISTENT- persistence directory node

The node still exists after the client is disconnected from the zookeeper

2. PERSISTENT_SEQUENTIAL- persistence sequentially numbered directory nodes

After the client is disconnected from zookeeper, the node still exists, but Zookeeper numbers the node name sequentially.

3. EPHEMERAL- temporary directory node

After the client is disconnected from zookeeper, the node is deleted

4. EPHEMERAL_SEQUENTIAL- temporary sequential numbering directory node

After the client is disconnected from the zookeeper, the node is deleted, but the Zookeeper numbers the node name sequentially

4.Zookeeper notification mechanism

The client registers to listen to the directory node it cares about, and zookeeper notifies the client when the directory node changes (data is changed, deleted, and subdirectory nodes are added and deleted).

What did 5.Zookeeper do?

1. Naming Service 2. Configuration Management 3. Cluster management 4. Distributed lock 5. Queue management

6.Zookeeper naming service

Create a directory in zookeeper's file system that has a unique path. When we can not determine the deployment machine of the upstream program using tborg, we can agree on path with the downstream program, and we can explore and discover each other through path.

Configuration Management of 7.Zookeeper

Programs always need to be configured, and if the programs are distributed across multiple machines, it becomes difficult to change the configuration one by one. Now put all these configurations on zookeeper and save them in a directory node in Zookeeper, and then all relevant applications listen to this directory node. Once the configuration information changes, each application will receive a notification from Zookeeper, and then get the new configuration information from Zookeeper and apply it to the system.

8.Zookeeper cluster management

The so-called cluster management does not care about two things: whether there are machines to quit and join, and to elect master.

For the first point, all machines agree to create a temporary directory node under the parent directory GroupMembers, and then listen for child node change messages of the parent directory node. Once a machine dies, the machine is disconnected from zookeeper, the temporary directory node it creates is deleted, and all other machines are notified that a sibling directory has been deleted, so everyone knows that it is on board.

The addition of new machines is similar, and all machines are notified: the new sibling directory has been added, and there is highcount again. For the second point, let's change it a little bit. All machines create temporary sequentially numbered directory nodes, and each time select the machine with the lowest number as the master.

9.Zookeeper distributed lock

With zookeeper's consistent file system, the locking problem becomes easier. Lock services can be divided into two categories, one is to keep exclusive, the other is to control the timing.

For the first category, we think of a znode on zookeeper as a lock, implemented by createznode. All clients create the / distribute_lock node, and the client that is successfully created owns the lock. The lock is released after deleting the distribute_lock node that you created.

For the second category, / distribute_lock already exists in advance, and all clients create temporary sequentially numbered directory nodes under it. Like selecting master, the one with the lowest number gets the lock and deletes it after use, which is convenient in turn.

10.Zookeeper queue management

Two types of queues:

1. Synchronize the queue, which is available only when the members of a queue are gathered together, otherwise all the members are waiting for them to arrive.

2. The queue operates in and out of the queue in accordance with the FIFO mode.

First, create temporary directory nodes under the agreed directory and listen to whether the number of nodes is the number we require.

The second category is consistent with the basic principle of the control timing scenario in the distributed lock service, numbered in the list and numbered in the dequeue.

11. Distribution and data replication

Zookeeper provides consistent data services as a cluster, and naturally, it replicates data between all machines. Benefits of data replication:

1. Fault tolerance: one node makes an error so that the whole system does not stop working, and other nodes can take over its work

2. Improve the scalability of the system: distribute the load to multiple nodes, or add nodes to improve the load capacity of the system.

3. Improve performance: let the client access the nearest node locally to improve the access speed of users.

From the perspective of the transparency of client read and write access, data replication cluster systems are divided into the following two types:

1. WriteMaster: changes to the data are submitted to the specified node. Read without this restriction, you can read any node. In this case, the client needs to distinguish between reading and writing, commonly known as read-write separation.

2. Write Any: modifications to the data can be submitted to any node, just like reading. In this case, the client is transparent to the roles and changes of the cluster nodes.

For zookeeper, it takes the approach of writing arbitrarily. By adding machines, its read throughput and response capacity is very scalable, while writing, as the number of machines increases, the throughput must decline (which is why it established observer), and the response ability depends on the specific implementation, whether to delay replication to maintain final consistency or immediate replication to respond quickly.

12.Zookeeper role description

13.Zookeeper and client

Design purpose of 14.Zookeeper

1. Ultimate consistency: no matter which Server client connects to, it is shown to be the same view, which is the most important performance of zookeeper.

two。 Reliability: with simple, robust, good performance, if the message is accepted by one server, then it will be accepted by all servers.

3. Real-time: Zookeeper ensures that the client will get the update information of the server or the information of server failure within a time interval. However, due to network delay and other reasons, Zookeeper can not guarantee that two clients can get the newly updated data at the same time. If the latest data is needed, the sync () interface should be called before reading the data.

4. Wait irrelevant (wait-free): slow or invalid client must not interfere with fast client requests so that each client can wait effectively.

5. Atomicity: updates can only succeed or fail, with no intermediate state.

6. Ordering: including global order and partial order: global order means that if message an is published before message b on a server, message a will be published before message b on all Server; partial order means that if a message b is released by the same sender after message a, a will be before b.

How 15.Zookeeper works

The core of Zookeeper is atomic broadcasting, which ensures synchronization between Server. The protocol that implements this mechanism is called the Zab protocol. There are two modes of Zab protocol, which are recovery mode (primary mode) and broadcast mode (synchronization). When the service starts or after the leader crashes, the Zab enters the recovery mode, and when the leader is elected and most of the Server finishes synchronizing with the leader, the recovery mode ends. State synchronization ensures that leader and Server have the same system state.

In order to ensure the order consistency of transactions, zookeeper uses an incremental transaction id number (zxid) to identify transactions. All proposals (proposal) are made with zxid when they are made. In the implementation, zxid is a 64-bit number, and its high 32-bit epoch is used to identify whether the leader relationship has changed. Each time a leader is selected, it will have a new epoch that identifies the current period of leader rule. The lower 32 bits are used to increment the count.

Working status of Server under 16.Zookeeper

Each Server has three states during its operation:

LOOKING: currently Server doesn't know who leader is and is searching for it.

LEADING: the current Server is the elected leader

FOLLOWING:leader has been elected, and the current Server is in sync with it

17.Zookeeper selection main process (basic paxos)

When leader crashes or leader loses most of its follower, the zk enters recovery mode, which requires a new leader to be re-elected so that all Server are restored to the correct state. There are two election algorithms for Zk: one is based on basic paxos, and the other is based on fast paxos algorithm. The default election algorithm of the system is fast paxos.

1. The election thread is held by the thread initiated by the current Server. Its main function is to count the voting results and select the recommended Server.

two。 The election thread first initiates a query to all Server (including itself)

3. After receiving the reply, the election thread verifies whether the query is initiated by itself (verify whether the zxid is consistent), then obtains the id (myid) of the other party, stores it in the list of current query objects, and finally obtains the leader-related information (id,zxid) proposed by the other party, and stores the information in the voting record table of the current election.

4. After receiving all the Server responses, calculate the Server with the largest zxid, and set the Server related information to the Server to vote next time.

5. The thread sets the maximum Server of the current zxid to the Leader recommended by the current Server. If the winning Server gets the number of Server votes of Server 2 + 1, and sets the currently recommended leader to the winning Server, it will set its own status according to the information about the winning Server, otherwise, the process will continue until the leader is elected. Through the process analysis, we can draw a conclusion: for Leader to be supported by most Server, the total number of Server must be odd 2n+1, and the number of surviving Server must not be less than 1. The above process is repeated after each Server is started. Under the recovery model, if you have just recovered from a crash or just started server will also restore data and session information from disk snapshots, zk will record transaction logs and take periodic snapshots to facilitate state recovery during recovery. The specific flow chart of the elector is shown as follows:

18.Zookeeper selection main process (fast paxos)

The fast paxos process is that in the election process, a Server first proposes to all Server that it wants to become a leader. When other Server receives the proposal, it resolves the conflict between epoch and zxid, accepts the proposal, and then sends a message accepting the proposal to the other party, repeating the process, and finally electing the Leader.

19.Zookeeper synchronization process

After selecting Leader, zk enters the state synchronization process.

1. Leader waits for server connection

2.Follower connects to leader and sends the largest zxid to leader.

3. Leader determines the synchronization point according to the zxid of follower

4. Notify follower that it has become a uptodate state after completing the synchronization

5. After receiving the uptodate message, Follower can accept the client request for service again.

20.Zookeeper Workflow-Leader

1. Recover data

2. Maintain the heartbeat with Learner, receive Learner requests and determine the request message type of Learner

3. The message types of Learner mainly include PING message, REQUEST message, ACK message and REVALIDATE message, which are processed differently according to different message types.

PING message refers to the heartbeat information of Learner.

REQUEST message is the proposal information sent by Follower, including write request and synchronization request

The ACK message is Follower's reply to the proposal, and if more than half of the Follower is approved, commit the proposal.

REVALIDATE messages are used to extend the validity of SESSION.

Supplement: transaction operation

The handling of transaction operations by the ZAB protocol is similar to a two-phase commit process. For the transaction request of the client, the leader server generates the corresponding transaction proposal for it, sends it to all follower machines in the cluster, then collects their votes, and finally commits the transaction. The process is shown in the following figure.

In the two-phase commit process of ZAB protocol, the interrupt logic (transaction rollback) is removed, and all follower servers either normally feedback the transaction proposal proposed by leader, or abandon the leader server. After follower receives the proposal, it is very simple to write the proposal to the transaction log, and then immediately feedback the ACK to the leader, that is, if it is not a network, memory or disk problem, the follower will write successfully and feedback the ACK normally. When leader receives more than half a follower of ACK, it broadcasts the commit message to all learner and applies the transaction to memory; when learner receives the commit message, it applies the transaction to memory.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.