Shulou(Shulou.com)06/03 Report--

This article introduces the relevant knowledge of "explaining the core knowledge points of zookeeper". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

First, first acquaintance of zookeeper?

Zookeeper, as an open source sub-project of the Hadoop project, is a classic distributed data consistency solution, which aims to provide a distributed coordination service with high performance, high availability and strict sequential access control for distributed applications.

1. Zookeeper data model

Zookeeper maintains a data structure similar to a file system, and each subdirectory (/ Wechat, / Wechat / official account) is called znode, or node. Like the file system, we can easily add or delete znode nodes, and we can also add or delete sub-znode under a znode. The difference is that znode can store data (strictly speaking, it must be stored, and the default is an empty character).

Because zookeeper is a directory node structure, you must start with "/" when getting and creating nodes, otherwise you will get an error Path must start with / character when getting nodes.

[zk: localhost:2181 (CONNECTED) 13] get testCommand failed: java.lang.IllegalArgumentException: Path must start with / character

The root node name must be "/ XXX", and the root node directories "/ XXX/CCC" and "/ XXX/AAA" must be taken when creating child nodes.

For example, in order to get the following picture, the programmer's internal things node must piece together the complete path get / Wechat / official account / programmer's internal things.

Something inside get / Wechat / official account / programmer

Znode is used to store byte-level or kb-level data, and the maximum amount of data that can be stored is 1MB (please note: the data amount of a node not only includes its own stored data, but also the names of all its child nodes should be converted into Byte, so the number of znode child nodes is not unlimited. Although the node storage size can be manually modified, this is not recommended in general.

2. Znode node attributes

A znode node can not only store data, but also have some other special properties. Next, let's create a / test node to analyze the meaning of its attributes.

[zk: localhost:2181 (CONNECTED) 6] get / test456cZxid = 0x59ac / / ctime = Mon Mar 30 15:20:08 CST 2020mZxid = 0x59admtime = Mon Mar 30 15:22:25 CST 2020pZxid = 0dataVersion = 2aclVersion = 0ephemeralOwner = 0x0dataLength = 3numChildren = 0 Node attribute Note cZxid the transaction when the data node is created IdmZxid the latest thing when the data node is modified the parent node transaction Idctime of the current node mtime the creation time of the data node mtime After modification time dataVersion current node version number (each modification value + 1 increment) cversion child node version number (number of child node modifications Each modification value + 1 increment) aclVersion current node acl version number (node is modified acl permission, each modification value + 1 increment) ephemeralOwner temporary node indicates that if the current node is a temporary node, the stored session id (sessionId) of the creator, if not, then the value = the length of data stored by the current node numChildren the number of child nodes under the current node

We can see that a znode node has more attributes, but the main attributes are zxid, version, and acl.

Zxid:

The state change of the znode node will cause the node to receive a timestamp in zxid format, which is globally ordered, and the establishment or update of the znode node will produce a new timestamp. If the value of zxid1

< zxid2的值，那么说明zxid2发生的改变在zxid1之后。每个znode节点都有3个zxid属性，cZxid（节点创建时间）、mZxid（该节点修改时间，与子节点无关）、pZxid（该节点或者该节点的子节点的最后一次创建或者修改时间，孙子节点无关）。 zxid属性主要应用于zookeeper的集群，这个后边介绍集群时详细说。 Version： znode属性中一共有三个版本号dataversion（数据版本号）、cversion（子节点版本号）、aclversion（节点所拥有的ACL权限版本号）。 znode中的数据可以有多个版本，如果某一个节点下存有多个数据版本，那么查询这个节点数据就需要带上版本号。每当我们对znode节点数据修改后，该节点的dataversion版本号会递增。当客户端请求该znode节点时，会同时返回节点数据和版本号。另外当dataversion为 -1的时候可以忽略版本进行操作。对一个节点设置权限时aclVersion版本号会递增，下边会详细说ACL权限控制。 验证一下，我们修改/test节点的数据看看dataVersion 有什么变化，发现dataVersion 属性变成了 3，版本号递增了。 [zk: localhost:2181(CONNECTED) 10] set /test 8888cZxid = 0x59acctime = Mon Mar 30 15:20:08 CST 2020mZxid = 0x59b6mtime = Mon Mar 30 16:58:08 CST 2020pZxid = 0x59accversion = 0dataVersion = 3aclVersion = 0ephemeralOwner = 0x0dataLength = 4numChildren = 03、znode的类型 zookeeper 有四种类型的znode，在用客户端 client 创建节点的时候需要指定类型。 zookeeper.create("/公众号/程序员内点事", "".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL); PERSISTENT-持久化目录节点 ：client创建节点后，与zookeeper断开连接该节点将被持久化，当client再次连接后节点依旧存在。 PERSISTENT_SEQUENTIAL-持久化顺序节点 ：client创建节点后，与zookeeper断开连接该节点将被持久化，再次连接节点还存在，zookeeper会给该节点名称进行顺序编号，例如：/lock/0000000001、/lock/0000000002、/lock/0000000003。 EPHEMERAL-临时目录节点 ： client与zookeeper断开连接后，该节点即会被删除 EPHEMERAL_SEQUENTIAL-临时顺序节点 ： client与zookeeper断开连接后，该节点被删除，会给该节点名称进行顺序编号，例如：/lock/0000000001、/lock/0000000002、/lock/0000000003。 二、节点的ACL权限控制 ACL：即 Access Control List (节点的权限控制)，通过ACL机制来解决znode节点的访问权限问题，要注意的是zookeeper对权限的控制是基于znode级别的，也就说节点之间的权限不具有继承性，即子节点不继承父节点的权限。 zookeeper中设置ACL权限的格式由::三段组成。 schema ：表示授权的方式 world：表示任何人都可以访问 auth：只有认证的用户可以访问 digest：使用username ：password用户密码生成MD5哈希值作为认证ID host/ip：使用客户端主机IP地址来进行认证 id： 权限的作用域，用来标识身份，依赖于schema选择哪种方式。 acl：给一个节点赋予哪些权限，节点的权限有create,、delete、write、read、admin 统称 cdwra。 1、world：表示任何人都可以访问 我们用 getAcl 命令来看一下，没有设置过权限的znode节点，默认情况下的权限情况。 [zk: localhost:2181(CONNECTED) 12] getAcl /test'world,'anyone: cdrwa 看到没有设置ACL属性的节点，默认schema 使用的是world，作用域是anyone，节点权限是cdwra，也就是说任何人都可以访问。 那我们如果要给一个schema 为非world的节点设置world权限咋搞？ setAcl /test world:anyone:crdwa2、auth：只有认证的用户可以访问 schema 用auth授权表示只有认证后的用户才可以访问，那么首先就需要添加认证用户，添加完以后需要对认证的用户设置ACL权限。 addauth digest test:password(明文) 需要注意的是设置认证用户时的密码是明文的。 [zk: localhost:2181(CONNECTED) 2] addauth digest user:user //用户名：密码[zk: localhost:2181(CONNECTED) 5] setAcl /test auth:user:crdwa[zk: localhost:2181(CONNECTED) 6] getAcl /test'digest,'user:ben+k/3JomjGj4mfd4fYsfM6p0A=: cdrwa 实际上我们这样设置以后，就是将这个节点开放给所有认证的用户，setAcl /test auth:user:crdwa 相当于setAcl /test auth::crdwa。 3、digest：用户名:密码的验证方式 用户名:密码方式授权是针对单个特定用户，这种方式是不需要先添加认证用户的。 如果在代码中使用zookeeper客户端设置ACL，那么密码是明文的，但若是zk.cli等客户端操作就需要将密码进行sha1及base64处理。 setAcl digest:::setAcl /test digest:user:jalRr+knv/6L2uXdenC93dEDNuE=:crdwa 那么密码如何加密嘞？有以下几种方式： 通过shell命令加密 echo -n : | openssl dgst -binary -sha1 | openssl base64 使用zookeeper自带的类库org.apache.zookeeper.server.auth.DigestAuthenticationProvider生成 java -cp /zookeeper-3.4.13/zookeeper-3.4.13.jar:/zookeeper-3.4.13/lib/slf4j-api-1.7.25.jar \ org.apache.zookeeper.server.auth.DigestAuthenticationProvider \ root:rootroot:root->

Root:qiTlqPLK7XM2ht3HMn02qRpkKIE=4, host/ip: use client host IP address for authentication

This approach is easier to understand, by authorizing a specific IP address or an IP segment.

[zk: localhost:2181 (CONNECTED) 3] setAcl / test0000000014 ip:127.0.0.1:crdwacZxid = 0x59acctime = Mon Mar 30 15:20:08 CST 2020mZxid = 0x59b6mtime = Mon Mar 30 16:58:08 CST 2020pZxid = 0x59accversion = 0dataVersion = 3aclVersion = 3 / / this version has been adding ephemeralOwner = 0x0dataLength = 4numChildren = 0 III, zookeeper's soul watcher

We said at the beginning: zookeeper can provide service registration and discovery for dubbo, as a registry, but have you ever wondered why zookeeper can achieve service registration and discovery? So I have to talk about the soul of zookeeper, Watcher (listener).

1. What is watcher?

Watcher is a very core function in zooKeeper. The client watcher can monitor the data changes of the node and the changes of its child nodes. Once these states change, the zooKeeper server will notify all the clients that have set watcher on this node, so that each client will quickly perceive that the state of the node it listens to has changed, and make corresponding logic processing.

After a brief introduction to watcher, let's analyze how zookeeper implements service registration and discovery. The service registration and discovery of zookeeper mainly applies zookeeper's znode node data model and watcher mechanism. The general process is as follows:

Service registration: when the service provider (Provider) starts, it registers the service information with the zookeeper server, that is, creates a node, for example, the user registers the service com.xxx.user.register, and stores the relevant data of the service (such as the ip address, port, etc.) of the service on the node.

Service discovery: when the service consumer (Consumer) starts, it obtains the registered service information from the zookeeper server and sets up watch snooping according to the dependent service information configured by itself. After obtaining the registered service information, it caches the information of the service provider locally and invokes the service.

Service notification: once the service provider ceases to provide services for some reason, the client disconnects from the zookeeper server, and the service node corresponding to the service provider on the zookeeper server will be deleted (for example, user registration service com.xxx.user.register), and then the zookeeper server will asynchronously register the service com.xxx.user.register with all consumers. And the service consumer with watch monitoring sends a notification that the node is deleted, and the consumer pulls the latest service list according to the notification received and updates the locally cached service list.

The above process is the general principle that zookeeper can implement service registration and discovery.

2. Watcher type

Znode nodes can set up two types of watch, one is DataWatches, which triggers watch events based on data changes of znode nodes, triggering conditions getData (), exists (), setData (), and create ().

The other is Child Watches, which triggers watch events triggered by changes in znode-based child nodes, triggering conditions getChildren () and create ().

When the delete () method is called to delete znode, both Data Watches and Child Watches are triggered, and if the deleted node has a parent node, the parent node triggers a Child Watches.

3. Watcher characteristics

The listening event of watch to the node is one-time! The client sets the listening watch on the specified node, and once the client is notified of a change in the data of the node, the listening event of the client to the node is invalid.

If we want to continue listening on this node, we need to set the listening watch event of the node to True again in the client's listening callback. Otherwise, the client can only receive a change notification from that node once.

IV. What functions can zookeeper achieve?

The registration and discovery function of the service is only the tip of the iceberg of zookeeper. It can also achieve a series of functions, such as distributed locks, queues, configuration centers, and so on. Next, we will only analyze the principle. The specific implementation is relatively complete for everyone to check the information on the Internet.

1. Distributed lock

Based on the watcher mechanism and the ordered node of znode, zookeeper is born as the blank of a distributed lock. First create a / test/lock parent node as a lock, try to be a persistent node (PERSISTENT type), and each client that tries to acquire the lock creates a temporary sequential child node under the / test/lock parent node.

Because of the increment of the sequence number, we stipulate that the node with the lowest sequence number acquires the lock. For example, the client acquires the lock and creates a node under the / test/lock node as / test/lock/seq-00000001, which is the smallest so it gets the lock first, and the other nodes wait for the notification to acquire the lock again. / test/lock/seq-00000001 deletes the node to release the lock after executing its own logic.

So whose notification does the node / test/lock/seq-00000002 want to get the lock and so on?

Here, we let the / test/lock/seq-00000002 node listen on the / test/lock/seq-00000001 node. Once the / test/lock/seq-00000001 node is deleted, notify the / test/lock/seq-00000002 node and let it determine again whether it is the smallest node. If so, it will get the lock instead of waiting for the notification.

And so on / test/lock/seq-00000003 node listening / test/lock/seq-00000002 node, always let the latter node listen to the previous node, do not have to let all nodes listen to the smallest node, avoid setting unnecessary monitoring, so as to avoid causing a large number of invalid notifications, resulting in a "herding effect".

Compared with redis distributed locks, zookeeper distributed locks are not recommended because of the poor performance of a large number of creating and deleting nodes.

2. Distributed queue

Zookeeper is also very simple to implement distributed queues, using the natural "first-in, first-out" of the ordered nodes of znode, the nodes created later are always the largest, and the nodes with the lowest sequence number are always available for queuing.

3. Configuration management

There are many open source projects that use Zookeeper to maintain configurations, such as message queuing Kafka, using Zookeeper to maintain broker information, and dubbo to manage service configuration information. The principle is also based on the watcher mechanism, for example: create a / config node to store some configurations, the client listens to this node, modifies the configuration information of the / config node, and notifies each client to re-pull the configuration information when the data is changed.

4. Naming service

Zookeeper naming service: what we often call service registration and discovery, mainly according to the specified name to obtain resource or service address, service provider and other information, making use of the characteristics of its znode node and watcher mechanism, it is used as a configuration center to dynamically register and obtain service information, and uniformly manage the service name and its corresponding server list information. We can sense the status of the back-end server (online, offline, downtime) in near real time.

This is the end of the introduction to the core knowledge points of zookeeper. Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.