Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about the common use scenarios of zookeeper, which may not be well understood by many people. in order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

ZooKeeper is a highly available distributed data management and system coordination framework. Based on the implementation of the Paxos algorithm, the framework ensures the strong consistency of data in the distributed environment, and it is based on this characteristic that zookeeper can be applied to many scenarios. There is also a lot of introduction to the use scene of zk on the Internet. This paper will systematically classify and introduce the use scene of zk combined with the project examples around the author. It is worth noting that zk is not born to design for these scenarios, but is a typical use that many developers have explored according to the characteristics of the framework. Therefore, you are also welcome to share your tricks and tricks in using ZK.

Scenario category typical scenario description (ZK feature, usage) specific use of data publish and subscribe publish and subscribe in the application is the so-called configuration management, as the name implies, is to publish the data to the zk node for subscribers to dynamically obtain data, to achieve centralized management and dynamic update of configuration information. For example, global configuration information, address lists, etc., are very suitable for use. 1. The index information and the state of the machine nodes in the cluster are stored in some designated nodes of the zk and are used by various clients to subscribe. two。 System logs (processed) are stored and are usually cleared after 2-3 days.

3. Some configuration information used in the application is centrally managed, which is actively obtained when the application is started, and a Watcher is registered on the node, and then each configuration update is notified to the application in real time to obtain the latest configuration information.

4. Some global variables that need to be used in the business logic, such as the message queue of some message middleware, usually have an offset, and this offset is stored on the zk, so that each sender in the cluster can know the current delivery progress.

5. Some information in the system needs to be obtained dynamically, and there will be manual manual modification of this information. In the past, interfaces are usually exposed, such as the JMX interface, but with zk, all you have to do is store this information on the zk node.

Name Service is mainly used as a distributed naming service. By calling zk's create node api, you can easily create a globally unique path, and this path can be used as a name.

Distributed notification / coordination ZooKeeper has a unique watcher registration and asynchronous notification mechanism, which can well realize the notification and coordination between different systems in the distributed environment, and realize the real-time processing of data changes. The method of use is usually that different systems register the same znode on the ZK and listen for changes in the znode (including the content of the znode itself and its child nodes). If one system update the znode, then the other system can receive the notification and deal with it accordingly. 1. Another heartbeat detection mechanism: the detection system and the detected system are not directly related, but through the association of a node on the zk, the system coupling is greatly reduced. two。 Another system scheduling mode: a system consists of a console and a push system, and the duty of the console is to control the push system to carry out the corresponding push work. Some of the operations that managers do in the console actually modify the status of some nodes on the ZK, and zk notifies them of these changes to the client that registers Watcher, that is, the push system, and then makes the corresponding push task.

3. Another work reporting mode: some similar to the task distribution system, after the sub-task starts, register a temporary node to zk, and report their progress regularly (write the progress back to this temporary node), so that the task manager can know the progress of the task in real time.

In short, using zookeeper for distributed notification and coordination can greatly reduce the coupling between systems.

Distributed lock distributed lock, this is mainly because ZooKeeper ensures the strong consistency of data for us, that is, as long as users fully believe that the data of the same znode on any node (a zk server) in the zk cluster must be the same all the time. Lock services can be divided into two categories, one is to keep exclusive, the other is to control the timing.

Maintaining exclusivity means that only one client who tries to acquire the lock can successfully acquire the lock. The usual practice is to treat a znode on zk as a lock, which is implemented by create znode. All clients create the / distribute_lock node, and the client that is successfully created owns the lock.

Control timing, that is, all the views to acquire the lock of the client, will eventually be scheduled to execute, but there is a global timing. The practice is similar to the above, except that / distribute_lock already exists here, and the client creates a temporary ordered node under it (this can be specified by the node's attribute control: CreateMode.EPHEMERAL_SEQUENTIAL). The parent node (/ distribute_lock) of the Zk maintains a copy of the sequence, which ensures the timing of the creation of the child nodes, thus forming the global timing of each client.

Cluster management 1. Cluster machine monitoring: this is usually used in scenarios with high requirements for the status and online rate of machines in the cluster, and can quickly respond to machine changes in the cluster. In such scenarios, there is often a monitoring system that detects the survival of cluster machines in real time. In the past, the monitoring system used to regularly detect each machine through some means (such as ping), or each machine regularly reported to the monitoring system, "I am still alive." This approach is feasible, but there are two obvious problems: 1. When there are changes in the machines in the cluster, there are more things that are involved and modified. two。 There is a certain delay.

Making use of the two characteristics of ZooKeeper, we can real-time another cluster machine viability monitoring system: a. The client registers a Watcher on node x and notifies the client if the child nodes of x change. b. Create a node of type EPHEMERAL that disappears once the client and server session ends or expires.

For example, the monitoring system registers a Watcher on the / clusterServers node and creates an EPHEMERAL node under / clusterServers: / clusterServers/ {hostname} each time the machine is added dynamically. In this way, the monitoring system can know the increase and decrease of the machine in real time, and the follow-up processing is the business of the monitoring system.

2. Master election is the most classic scenario in zookeeper.

In a distributed environment, the same business applications are distributed on different machines, and some business logic (for example, some time-consuming computing, network I and O processing) often only need to be executed by one machine in the whole cluster, and the rest of the machines can share this result, which can greatly reduce repetitive work and improve performance, so the master election is the main problem encountered in this scenario.

Using the strong consistency of ZooKeeper, we can ensure the global uniqueness of node creation in the case of distributed high concurrency, that is, there are multiple client requests to create / currentMaster nodes at the same time, and only one client request can be created successfully.

With this feature, cluster selection can be easily carried out in a distributed environment.

In addition, this scenario evolves to be dynamic Master elections. This is going to use the features of the EPHEMERAL_SEQUENTIAL type node.

As mentioned above, only one of all client creation requests can be created successfully. With a slight change here, all requests are allowed to be created successfully, but there must be a creation order, so there is a possibility that all requests will eventually create results on ZK: / currentMaster/ {sessionId}-1, / currentMaster/ {sessionId}-2, / currentMaster/ {sessionId}-3... .. Every time the machine with the lowest serial number is selected as the Master, if the machine dies, because the node it creates will be hourly immediately, then the smallest machine after that will be Master.

1. In the search system, if each machine in the cluster generates a full index, it is not only time-consuming, but also can not guarantee that the index data is consistent with each other. So let the Master in the cluster generate the full index and then synchronize it to other machines in the cluster. two。 In addition, the disaster recovery measure of Master election is that master can be specified manually at any time, that is, when zk is unable to obtain master information, it can obtain master from a place through, for example, http. At present, I feel that there are two kinds of distributed queues, one is the conventional first-in-first-out queue, and the other is to wait for the queue members to gather before they are executed in sequence. For the second FIFO queue, which is consistent with the basic principle of the control timing scenario in the distributed lock service, I will not repeat it here.

The second queue is actually an enhancement based on the FIFO queue. Usually, you can set up a / queue/num node in advance under the znode of / queue, and assign a value of n (or directly assign n to / queue) to indicate the queue size. After that, each time a queue member joins, you can determine whether the queue size has reached the queue size and decide whether execution can begin. A typical scenario for this usage is that in a distributed environment, a large task Task A can only be done when many subtasks are completed (or conditionally ready). At this time, when one of the subtasks is completed (ready), then set up your own temporary timing node (CreateMode.EPHEMERAL_SEQUENTIAL) under / taskList. When / taskList finds that the number of child nodes below it meets the specified number, it can proceed to the next step in sequence processing.

After reading the above, do you have any further understanding of the common usage scenarios of zookeeper? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.