Shulou(Shulou.com)06/01 Report--

This article shows you what the operation and application scenario of zookeeper is. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

As a distributed coordination system, Zookeeper provides a basic service: distributed lock service. Distributed lock is the core content of distributed coordination technology. Such as configuration management, task distribution, group service, distributed message queue, distributed notification / coordination and so on, these applications are actually based on this basic service.

Common Application of 1.Zookeeper in big data system

As a distributed coordination system, zookeeper is very commonly used in the field of big data, and it is a good centralized management tool. Here are a few common application scenarios.

1.1.HDFS/YARN

HA mechanism (application of distributed locks): quickly switch to the slave node after the Master is hung up.

1.2.hbase

HA mechanism: similar to HDFS/YARN.

Configuration management: the data that client needs to read and write to hbase is first connected to ZK to read the root table, get the region where the meta table is located, and finally find the location of the data.

Task release: regionserver is dead, master needs to reassign region, and will place the task in regionserver such as zookeeper to get it.

1.3.kafka

Configuration management: broker will register with zookeeper and keep relevant metadata (topic,partition information, etc.) updated

Task assignment: assign partitions and replication to topic

What are the operating features of 2.Zookeeper 2.1. Data structure

Znode in the ZooKeeper namespace has both file and directory characteristics. It not only maintains data, meta-information, ACL, timestamp and other data structures like files, but also can be used as part of path identification like directories. Each Znode consists of three parts:

Stat status information: describes the version and permissions of the Znode

Data: data associated with the Znode (profile information, status information, collection location), up to 1m in size

Children: the child node under the Znode

The data stored by each node in ZooKeeper is manipulated atomically. In other words, the read operation will get all the data related to the node, and the write operation will replace all the data of the node. In addition, each node has its own ACL (access Control list), which defines the permissions of the user, that is, the actions that a specific user can perform on the target node.

2.2.watch mechanism

ZooKeeper can set watch for all read operations, including exists (), getChildren (), and getData (). When the state of the node changes (the addition, deletion and modification of Znode), the operation corresponding to watch will be triggered. When the watch is triggered, the ZooKeeper will send only one notification to the client, because the watch can only be triggered once, which reduces network traffic.

Data watch (data watches): getData and exists are responsible for setting up data watch

Child watch (child watches): getChildren is responsible for setting the child's watch

2.3. Node Typ

There are two kinds of nodes in ZooKeeper: temporary nodes and permanent nodes (which can be subdivided into ordered and unordered nodes). The type of node is determined at creation time and cannot be changed.

Temporary nodes: the life cycle of this node depends on the session in which they were created. Once the session (Session) ends, the temporary node is automatically deleted, either manually or manually. Although each temporary Znode is bound to a client session, they are still visible to all clients. In addition, temporary nodes of ZooKeeper are not allowed to have child nodes. (distributed queue)

Permanent nodes: the life cycle of this node does not depend on the session, and they can be deleted only if the client shows that the delete operation is performed.

3. How do these applications implement the 3.1.HA mechanism through these features:

There are two ways:

Create two or more ordered temporary nodes and always treat the minimum as a master

Create a temporary node as master, and multiple slave will watch this node

3.2. Configuration Management:

The storage cluster metadata is provided to client for use, for example, when you need to operate on HBase and Kafka, it will be directly connected to zookeeper,zookeeper to record the location of data storage, survival nodes and other metadata information.

3.3. Task release:

Master monitors the two permanent nodes / works and / tasks so that it can sense which slave is currently available and that there are new tasks to be assigned. Allocation process: create the currently available workA under / assign, find the taskA to be assigned, and create / assign/workA/taskA

Zookeeper has a lot of similar applications, most of which are based on the above features. In general, zk is just a system that provides some features based on which users define their own usage.

The above content is what is the operation and application scenario of zookeeper. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.