Shulou(Shulou.com)06/03 Report--

This article introduces the knowledge of "what are the common questions in Zookeeper interview". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

NO1: what is zookeeper?

ZooKeeper is a distributed, open source distributed application coordination service, which is an open source implementation of Google's Chubby (Chubby is not open source). It is the manager of the cluster, monitoring the status of each node in the cluster and taking the next reasonable operation according to the feedback submitted by the node. Finally, the easy-to-use interface and the system with efficient performance and stable function are provided to the user.

One of the most common use scenarios of Zookeeper is to serve as a registry center for service producers and service consumers. Service producers register the services they provide with the Zookeeper center. When making service calls, service consumers first look up the services in Zookeeper, obtain the details of the service producers, and then call the content and data of the service producers. A simple example is shown below:

NO2: do you know the system architecture of Zookeeper?

The main things we need to know and master in the architecture diagram of ZooKeeper are:

(1) ZooKeeper is divided into server (Server) and client (Client), and the client can connect to any server of the entire ZooKeeper service (unless the leaderServes parameter is explicitly set, leader does not allow client connections).

(2) the client uses and maintains a TCP connection through which to send requests, receive responses, obtain observed events, and send information. If this TCP connection is interrupted, the client will automatically attempt to connect to another ZooKeeper server. The first time a client connects to the ZooKeeper service, the ZooKeeper server that can accept the connection establishes a session for the client. When the client connects to another server, the session is re-established by the new server.

(3) each Server in the above figure represents a machine that installs Zookeeper services, that is, the whole cluster providing Zookeeper services (or composed of pseudo clusters).

(4) the servers that make up the ZooKeeper service must know each other. They maintain a state image in memory, as well as transaction logs and snapshots in persistent storage, and the ZooKeeper service is available as long as most servers are available.

(5) when ZooKeeper starts, a leader,Leader will be selected from the instance to handle data updates and other operations. A successful update operation is marked if and only if most Server successfully modify data in memory. Each Server stores a piece of data in memory.

(6) Zookeeper can be replicated in clusters, and data consistency is maintained through Zab protocol (Zookeeper Atomic Broadcast) among clusters.

(7) Zab protocol consists of two stages: leader election phase and Atomic Brodcast phase.

A) A leader will be elected in the cluster, and the other machines will be called follower. All writes will be passed to leader and all updates will be communicated to follower through brodcast.

B) when leader crashes or leader loses most of its follower, a new leader needs to be re-elected so that all servers return to the correct state.

C) when leader is elected and most servers have finished synchronizing with the status of leader, the leadder election process is over and will enter the Atomic brodcast process.

D) Atomic Brodcast synchronizes the information between leader and follower to ensure that leader and follower have the same system state.

NO3: can you tell me how Zookeeper works?

The core of Zookeeper is atomic broadcasting, which ensures synchronization between Server. The protocol that implements this mechanism is called the Zab protocol.

There are two modes of Zab protocol, which are recovery mode (primary mode) and broadcast mode (synchronization).

The full name of Zab protocol is Zookeeper Atomic Broadcast** (Zookeeper Atomic broadcast). Zookeeper ensures the final consistency of distributed transactions through the Zab protocol. The Zab protocol requires that each Leader go through three stages: discovery, synchronization, and broadcasting.

When the service starts or after the leader crashes, the Zab enters the recovery mode, and when the leader is elected and most of the Server finishes synchronizing with the leader, the recovery mode ends. State synchronization ensures that leader and Server have the same system state.

In order to ensure the order consistency of transactions, zookeeper uses an incremental transaction id number (zxid) to identify transactions. All proposals (proposal) are added with zxid when they are put forward. In the implementation, zxid is a 64-bit number, and its high 32-bit epoch is used to identify whether the leader relationship has changed. Each time a leader is selected, it will have a new epoch that identifies the current reign of that leader. The 32nd bit is used for incremental counting.

Epoch: it can be understood as the name of the emperor. When the new emperor leader comes into being, there will be a new epoch name.

Each Server has three states during its operation:

LOOKING: currently Server doesn't know who leader is and is searching for it.

LEADING: the current Server is the elected leader.

FOLLOWING:leader has been elected and the current Server is synchronized with it.

Why is NO4:Zookeeper designed like this?

ZooKeeper is designed to provide distributed coordination services with high performance, high availability and sequential consistency, and to ensure the ultimate consistency of data.

High performance (simple data model)

Using tree structure to organize data nodes

All data nodes are stored in memory

Follower and Observer directly handle non-transactional requests

High availability (build cluster)

If more than half of the machines survive, the service will run normally.

Automatic Leader election

Order consistency (order of transaction operations)

Each transaction request is forwarded to Leader for processing

For each transaction, a globally unique incremental id (zxid,64 bit: epoch + self-incrementing id) is assigned

Final consistency

By proposing voting, the reliability of transaction submission is guaranteed.

The proposed voting method can only ensure that more than half of the nodes can see the latest data after Client receives the transaction and commits the transaction successfully.

NO5: do you know what characters are in Zookeeper?

System model:

Leader (leader)

The Leader server provides read and write services for the client. Responsible for voting initiation and resolution, update system status.

Learner (learner)

The follower (follower) Follower server provides read services for the client, participates in the Leader election process, and participates in the write operation "more than half write success" strategy.

The observer Observer server provides read services for the client, does not participate in the Leader election process, and does not participate in the write operation "more than half write success" strategy. It is used to improve the read performance of the cluster without affecting the write performance.

Client: the initiator of the service request.

NO6: are you familiar with Zookeeper node ZNode and related attributes?

What are the types of nodes?

There are two types of Znode:

Persistent: after the client and server are disconnected, the nodes created are not deleted (default).

Ephemeral: after the client and server are disconnected, the created node is deleted by itself.

There are four forms of Znode:

Persistent directory node (PERSISTENT): after the client is disconnected from Zookeeper, the node still has a persistence sequentially numbered directory node (PERSISTENT_SEQUENTIAL)

After the client is disconnected from Zookeeper, the node still exists, but Zookeeper numbers the node name sequentially: temporary directory node (EPHEMERAL)

After the client is disconnected from Zookeeper, the node is deleted: temporary sequentially numbered directory node (EPHEMERAL_SEQUENTIAL)

After the client is disconnected from the Zookeeper, the node is deleted, but the Zookeeper numbers the node name sequentially

"Note": set the sequence identity when you create the ZNode, append a value to the ZNode name, and the sequence number is a monotonously increasing counter maintained by the parent node.

What are the attributes of the node

A znode node can not only store data, but also have some other special properties. Next, let's create a / test node to analyze the meaning of its attributes.

[zk: localhost:2181 (CONNECTED) 6] get / test 456 cZxid = 0x59ac / / ctime = Mon Mar 30 15:20:08 CST 2020 mZxid = 0x59ad mtime = Mon Mar 30 15:22:25 CST 2020 pZxid = 0x59ac cversion = 0 dataVersion = 2 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 3 numChildren = 0

Attribute description

NO7: please briefly describe the main selection process of Zookeeper

The core of Zookeeper is atomic broadcasting, which ensures synchronization between Server. The protocol that implements this mechanism is called the Zab protocol. There are two modes of Zab protocol, which are recovery mode (primary mode) and broadcast mode (synchronization). When the service starts or after the leader crashes, the Zab enters the recovery mode, and when the leader is elected and most of the Server finishes synchronizing with the leader, the recovery mode ends. State synchronization ensures that leader and Server have the same system state. Leader election is the key to ensure the consistency of distributed data.

There are mainly two scenarios in which an election occurs: initialization and leader is not available.

The leader election begins when one of the following two situations occurs on a server in the zk cluster.

Server initialization starts.

The server cannot stay connected to the leader while it is running.

When a machine enters the leader election process, the current cluster may also be in the following two states.

A leader already exists in the cluster.

It is true that leader does not exist in the cluster.

First of all, in the first case, usually a machine in the cluster starts relatively late, and before it starts, the cluster is already working properly, that is, a leader server already exists. When the machine tries to elect a leader, it is told the leader information of the current server, and it only needs to establish a connection with the leader machine and synchronize the status.

The key point is that leader is no longer available, and the selection system at this time.

The voting information contains two basic pieces of information.

Sid: server id, which is used to identify the machine sequence number of the machine in the cluster.

Zxid: the zookeeper transaction id number.

Every change in the state of ZooKeeper corresponds to an increasing Transaction id, the id is called zxid., due to the increasing nature of zxid, if zxid1 is less than zxid2, then zxid1 must occur before zxid2. Creating any node, or updating the data of any node, or deleting any node will cause the state of Zookeeper to change, resulting in an increase in the value of zxid.

Identify a voting information in the form of (sid,zxid).

For example, if the current server wants to recommend a server with a sid of 1 and a zxid of 8 to be called leader, then the voting information can be expressed as (1)

After each machine in the cluster sends its own vote, it also accepts votes from other machines in the cluster. Each machine will process the votes received by other machines according to certain rules to decide whether it needs to change its own vote.

The rules are as follows:

In the initial stage, they will vote for themselves.

When you receive votes from other servers, you need to compete other people's votes with your own votes. The rules are as follows:

Check zxid first. Servers with larger zxid are preferred as leader. If the zxid is the same, compare the servers with larger sid,sid as leader.

NO8: have you ever understood the watch mechanism?

To put it simply: the client will register a watcher event to a znode. When the znode changes, these client will be notified by the ZooKeeper, and then the client can make business changes according to the znode changes.

Classic usage scenario: zookeeper provides service registration and discovery for dubbo, as a registry, but have you ever wondered why zookeeper can achieve service registration and discovery?

So I have to talk about the soul of zookeeper, Watcher (listener).

What is watcher?

Watcher is a very core function in zooKeeper. The client watcher can monitor the data changes of the node and the changes of its child nodes. Once these states change, the zooKeeper server will notify all the clients that have set watcher on this node, so that each client will quickly perceive that the state of the node it listens to has changed, and make corresponding logic processing.

After a brief introduction to watcher, let's analyze how zookeeper implements service registration and discovery. The service registration and discovery of zookeeper mainly applies zookeeper's znode node data model and watcher mechanism. The general process is as follows:

Service registration: when the service provider (Provider) starts, it registers the service information with the zookeeper server, that is, creates a node, for example, the user registers the service com.xxx.user.register, and stores the relevant data of the service (such as the ip address, port, etc.) of the service on the node.

Service discovery: when the service consumer (Consumer) starts, it obtains the registered service information from the zookeeper server and sets up watch snooping according to the dependent service information configured by itself. After obtaining the registered service information, it caches the information of the service provider locally and invokes the service.

Service notification: once the service provider stops providing services due to some reason, the client is disconnected from the zookeeper server, and the service node corresponding to the service provider on the zookeeper server is deleted (for example, user registration service com.xxx.user.register). Then the zookeeper server asynchronously registers the service com.xxx.user.register to all consumers, and the service consumer with watch monitoring is set up to notify the service consumer that the node has been deleted. The consumer pulls the latest service list according to the notification received and updates the locally cached service list.

The above process is the general principle that zookeeper can implement service registration and discovery.

What are the types of watcher?

Znode nodes can set up two types of watch, one is DataWatches, which triggers watch events based on data changes of znode nodes, triggering conditions getData (), exists (), setData (), and create ().

The other is Child Watches, which triggers watch events triggered by changes in znode-based child nodes, triggering conditions getChildren () and create ().

When the delete () method is called to delete znode, both Data Watches and Child Watches are triggered, and if the deleted node has a parent node, the parent node triggers a Child Watches.

What are the features of watcher?

The listening event of watch to the node is one-time! The client sets the listening watch on the specified node, and once the client is notified of a change in the data of the node, the listening event of the client to the node is invalid.

If we want to continue listening on this node, we need to set the listening watch event of the node to True again in the client's listening callback. Otherwise, the client can only receive a change notification from that node once.

NO9: what are the application scenarios of Zookeeper?

Data publishing and subscription

Publish and subscribe is the so-called configuration management, which, as the name implies, publishes the data to the ZooKeeper node for subscribers to obtain data dynamically, and realizes the centralized management and dynamic update of configuration information. For example, global configuration information, address lists, etc., are very suitable for use.

A common scenario of data publishing / subscription is the configuration center, where the publisher publishes the data to one or a series of nodes of the ZooKeeper for subscribers to subscribe to data to achieve the purpose of obtaining data dynamically.

Configuration information generally has several characteristics:

KV with small amount of data

Data content changes dynamically at run time

Cluster machines are shared with consistent configuration

ZooKeeper uses a combination of push and pull.

Push: the server will push the Wathcer event notification to the client that has registered the monitoring node

A: after receiving the notification, the client takes the initiative to pull the latest data from the server.

Naming service

As a distributed naming service, naming service refers to obtaining the address of a resource or service through a specified name. Using ZooKeeper to create a global path, the path can be used as a name to point to the cluster in the cluster, the address of the service provided, or a remote object, and so on.

The naming structure diagram of the unified naming service is as follows:

1. In a distributed environment, it is often necessary to name applications / services uniformly to facilitate the identification of different services.

Similar to the corresponding relationship between domain name and IP, IP is not easy to remember, while domain name is easy to remember.

Get the address, provider and other information of a resource or service by name.

2. Organize the service / application name according to the hierarchical structure.

The service name and address information can be written to ZooKeeper, and the client can obtain the list class of available services through ZooKeeper.

Configuration management

The program is distributed and deployed on different machines, and the configuration information of the program is placed under the znode of ZooKeeper. When there is a configuration change, that is, when the znode changes, you can change the configuration by changing the content of a directory node in the zk and notifying each client using watch.

The ZooKeeper configuration management structure diagram is as follows:

1. In distributed environment, configuration file management and synchronization is a common problem.

In a cluster, the configuration information of all nodes is the same, such as the Hadoop cluster.

After modifying the configuration file, you want to be able to quickly synchronize to each node.

2. Configuration management can be implemented by ZooKeeper.

Configuration information can be written to a Znode on ZooKeeper.

Each node listens to the Znode.

Once the data in the Znode is modified, the ZooKeeper notifies each node.

Cluster management

The so-called cluster management is: whether there are machines to exit and join, to elect master.

Cluster management mainly includes two aspects: cluster monitoring and cluster control. The former focuses on the collection of the running state of the cluster, while the latter operates and controls the cluster. In development and operation and maintenance, in the face of clusters, there are often the following requirements:

I want to know how many machines are working in the cluster.

Collect data on the runtime status of each machine in the cluster

Perform online and offline operations on the machines in the cluster

The cluster management structure diagram is as follows:

In the distributed environment, it is necessary to grasp the state of each node in real time, and some adjustments can be made according to the real-time state of the node.

It can be implemented by ZooKeeper.

Node information can be written to a Znode on ZooKeeper.

Listen to this Znode for real-time status changes.

3. Typical applications

Master status monitoring and election in Hbase.

With the strong consistency of ZooKeeper, the global uniqueness of node creation in the case of distributed high concurrency can be guaranteed, that is, there are multiple client requests to create / currentMaster nodes at the same time, and finally only one client request can be created successfully.

Distributed notification and coordination

1. In a distributed environment, there is often a service that needs to know the status of the sub-services it manages.

A) NameNode needs to know the status of each Datanode.

B) JobTracker needs to know the status of each TaskTracker.

2. The heartbeat detection mechanism can be realized by ZooKeeper.

3. Information push can be implemented by ZooKeeper, and ZooKeeper is equivalent to a publish / subscribe system.

Distributed lock

Different services on different nodes may need to access some resources sequentially, which requires a distributed lock.

Distributed lock has the following characteristics: write lock, read lock, temporal lock.

Write lock: a temporary unnumbered node created on the zk. Because it is unordered, it is not automatically numbered when it is created, so that only one client can get the lock and then write.

Read lock: create a temporary numbered node on the zk so that even the next time a client joins and creates the same node at the same time, he will automatically number the lock object and read it.

Sequential lock: a temporary numbered node created on the zk controls the lock according to the size of the number.

Distributed queue

There are two types of distributed queues:

1. When all the members of a queue are gathered together, the queue is available, otherwise all the members are waiting for it to arrive. This is a synchronous queue.

A) A job consists of multiple task, and the job runs only after all tasks are completed.

B) you can create a / job directory for job, and then create a temporary Znode for each completed task. Once the number of temporary nodes reaches the total number of task, it indicates that the job is running.

2. Queues perform queuing and dequeuing operations according to FIFO, such as implementing producer and consumer models.

NO10: do you know how listeners work?

Hongmeng official Strategic Cooperation to build HarmonyOS Technology Community

Create a Main () thread.

Create two threads in the Main () thread, one responsible for network connection communication (connect) and one for listening (listener).

The registered listening event is sent to the Zookeeper through the connect thread.

Add registered listening events to the list of registered listeners in Zookeeper.

When Zookeeper detects a change in data or path, it sends this message to the listener thread.

The process () method is called inside the Listener thread.

NO11: why is the number of Zookeeper clusters usually odd?

First of all, we need to clarify the rules of zookeeper election: leader election, which requires the number of available nodes > the total number of nodes / 2.

For example, marking whether a write is successful or not is not considered valid until more than half of the nodes send a write request. Similarly, Zookeeper selects the leader node only when more than half of the nodes agree. Finally, whether the Zookeeper is normal depends on whether more than half of the nodes are normal. This is based on the consistency principle of CAP.

Zookeeper has such a feature: as long as more than half of the machines in the cluster are working properly, then the whole cluster is available externally.

That is to say, if there are two zookeeper, as long as there is one dead zookeeper, the zookeeper cannot be used. Because 1 is not more than half, the death tolerance of the two zookeeper is 0.

By the same token, if there are three zookeeper and one is dead, there are still two normal ones left, more than half of them, so the tolerance of three zookeeper is 1.

By the same token:

2-> 0; two zookeeper and a maximum of 0 zookeeper can be unavailable.

3-> 1; three zookeeper and up to one zookeeper can be unavailable.

4-> 1; four zookeeper and up to one zookeeper can be unavailable.

5-> 2; five zookeeper and up to two zookeeper can be unavailable.

6-> 2; two zookeeper and a maximum of 0 zookeeper can be unavailable.

....

You will find a rule that the tolerance of 2n and 2n-1 is the same, both NMMI 1, so why add an unnecessary zookeeper in order to be more efficient.

Zookeeper's election strategy also requires the consent of more than half of the nodes to be elected leader, if it is an even number of nodes may lead to the same number of votes.

This is the end of the content of "what are the common questions in the Zookeeper interview". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.