Shulou(Shulou.com)06/03 Report--

This article introduces the knowledge of "how to use four-word command and Admin Server". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. JMX

> JMX (Java Management Extensions, that is, Java Management extension) is a framework that embeds management functions for applications, devices, systems, and so on. JMX can span a series of heterogeneous operating system platforms, system architectures and network transport protocols, and flexibly develop seamless integrated system, network and service management applications.

Don't you understand? If you have never developed JMX or an application, I will briefly introduce it here: JMX is a standard provided by Java, which can expose the runtime status of some needed Java objects (these objects can be called MBean) and are generally used to monitor or modify some configuration information at run time.

Since we are talking about ZK, I have a problem now. Now that I have a simple ZK cluster running, I want to know which node is Leader. What should I do?

First of all, ZK itself takes the initiative to register some objects as MBean when it starts, and we can view them directly using jconsole, a tool that comes with Java. Let me demonstrate below that I have a simple ZK cluster here:

$jps5266 QuorumPeerMain296410438 Jps4550 Launcher5286 QuorumPeerMain5767 JConsole5388 QuorumPeerMain9215 Launcher

You can see that there are three QuorumPeerMain processes that represent the three nodes of the cluster.

Let's use jconsole to open the tool (once jdk is installed, the tool will have it automatically)

$jconsole

The figure shows that the operation can be called by a button on the right (you can also pass parameters). I will not disclose more details about JMX in ZK for the time being, and then I will have the opportunity to explain it separately. Anyway, we only need to know that JMX is to expose some ordinary Java objects, and you can use tools to view properties or call the object's methods.

Two-word, four-word command and Admin Server

Just now the JMX view is still quite troublesome, because now we are testing to access my local process, if it is a remote JVM process, it is even more troublesome to access it with jconsole, is there a simpler way. There must be! ZK itself supports some four-word commands (4lw) for interacting with the server.

Let me make a simple demonstration here. The client ports of my local cluster are 2181, 2182 and 2183 respectively. I can connect to any node casually through the telnet command:

Telnet localhost 2181Trying:: 1...Connected to localhost.Escape character is'^]'.

You will enter interactive mode, then type srvr and press enter to get the following output

$telnet localhost 2181Trying:: 1...Connected to localhost.Escape character is'^]'. SrvrZookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09 take 04 29Connections 12:44 GMTLatency min/avg/max: 0/3.8261/44Received: 30Sent: 29Connections: 1Outstanding: 0Zxid: 0x100000009Mode: leaderNode count: 6Proposal sizes last/min/max: 48/48/94Connection closed by foreign host.

The srvr command is used to check the status of the service node. From the Mode field in the output, you can see that the node listening to port 2181 is Leader. Let's change to another 2182 node, and Mode is Follower.

$telnet localhost 2182Trying:: 1...Connected to localhost.Escape character is'^]'. SrvrZookeeper version: 3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09 take 04 3Connections 12:44 GMTLatency min/avg/max: 0/0.0/0Received: 3Sent: 3Connections: 1Outstanding: 0Zxid: 0x100000009Mode: followerNode count: 6Connection closed by foreign host.

I would like to mention here that not all the default four-word commands are open, and if you want to enable all four-word commands, you need to specify zookeeper.4lw.commands.whitelist=*, in the environment variable or you can enable some of the specified four-word commands by listing specific commands (separated by commas). Let's try another command, such as envi, which will output the environment parameters of the current node.

Java.io.tmpdir=/var/folders/19/bx8xsqgd1c78g5j1mt_zq5v80000gp/T/java.compiler=os.name=Mac OS Xos.arch=x86_64os.version=10.16user.name=junjiexunuser.home=/Users/junjiexunuser.dir=/Users/junjiexun/develop/zkos.memory.free=101MBos.memory.max=889MBos.memory.total=123MB

Let me list the functions of all the four-word commands here, but I won't demonstrate them. I'll leave it to the reader to try. The four-word command list on the official website is actually just an alias for the command on the right, and the function is exactly the same. (* still means TODO, and then I'll start a monologue)

Command returns the role of data conf / configuration configuration information (commonly used dataDir, clientPort, etc.) cons / connections connection information crst / connection_stat_reset reset connection statistics dump session information and temporary node envi / environment environment variable information (os.name, User.home, etc.) whether the ruok service is normal srst / stat_reset reset statistics srvr / server_stats server information overview stat server information statistics wchs / watch_summary callback watcher summary wchc / watches registration callback session information summary dirslog and snap file byte size wchp / watches_by_path registration callback path information mntr / monitor all monitoring information isro / is_read_only whether the current node is read-only hash digits Abstract gtmk / get_trace_mask* get trace mask stmk / set_trace_mask* set trace mask lsnp / last_snapshot information of last snapshot icfg / initial_configuration server starts initial configuration orst / observer_connection_stat_reset reset Observer connection statistics obsr / observers get Observer information sysp / system_properties environment variable information is different from envi in that it returns the custom configuration lead / leader current section that begins with zookeeper Whether the point is a summary of Leadervoting_view cluster ballot information zabstateZAB status information

I also saw ruok, which is used to check whether the server node is started (whether the server node can be returned successfully does not mean it can provide services)

If you access these hyperlinks directly, you can have the same effect as the previous four-word command. You can also access ip:port/commands/ directly on URL.

Take mntr as an example, it is the same that you can access http://localhost:8080/commands/mntr or http://localhost:8080/commands/monitor directly.

3.1 vote counting rules

Before that, let me introduce two kinds of vote counting rules supported by ZK.

3.1.1 more than half mechanism

This is the default vote counting rule of ZK. It is used in scenarios where ACK is required for various server clusters, assuming that the current configuration is like this:

Server.1=zoo1:2888:3888:participantserver.2=zoo2:2888:3888:participantserver.3=zoo3:2888:3888:participantserver.4=zoo4:2888:3888:observerserver.5=zoo5:2888:3888:observer

Because Observer will not be counted as votes, the actual participating machines are the first three nodes: 1, 2, 3

No matter who the Leader is, the default counting rule requires at least two of the three nodes to successfully submit the ACK (or other information that requires counting) before the election (or proposal) can be submitted again. This is the more than half mechanism.

3.1.2 packet weight

ZK also provides a new vote counting rule, which supports dividing each node into different groups (of course, there can be only one group), and different nodes in the same group can also be assigned different weights. Let me give you an example:

Group.1=1:2:3group.2=4:5:6:7:8group.3=9weight.1=1weight.2=1weight.3=1weight.4=1weight.5=1weight.6=1weight.7=1weight.8=1weight.9=1

The beginning of group and the beginning of weight correspond to the configuration of grouping and weight, respectively, as follows:

The format of group is group.=:...

The format of weight is weight.=

ServerId is the number of each service node configured in the myid

Each node can only belong to one group

If we continue to explain in the case of my current configuration, there are now a total of three group, and their weights are calculated as follows:

Group1's weight sum = server1's weight + server2's weight + server3's weight = 1 + 1 + 1 = 3group2's weight sum = server4's weight + server5's weight + server6's weight + server8's weight = 1 + 1 + 1 + 1 + 1 = 5group3's weight sum = server9's weight = 1

If the service nodes of the successful ACK have 1, 4, 5, 8, 9 and count the votes with this configuration

First of all, look at group1, only server1 successfully replied to ACK, and the weight value is 1, which is not more than 3% of the total weight of group1, so group1 is equivalent to ACK failed.

Group2 has 4, 5 and 8 nodes successfully replying to ACK, and the weight value is 3, which is more than half of the total weight of group2 (3 > 5 group2 2), so group2 ACK is successful.

Then look at group3, because there is only one node 9, and successfully replied to ACK, so it also satisfies more than half of the total weight of group3 (1 > 1), so group3 ACK is successful

Finally, it is counted whether the number of group of successful ACK is more than half of the total number of group. Now there are 2 group successfully ACK (2 > 3 ACK 2), so the final ACK is passed.

It feels like the default more than half mechanism, No,No,No. The reason I look the same here is because I set the weight to 1. What if I set it to another number?

In the scenario just now, only group1's ACK failed, because only one node of server1 successfully replied, but if I changed the weight of group1 to (the other two group were omitted)

Group.1=1:2:3weight.1=3weight.2=1weight.3=1

Now the total weight of group1 is 3 + 1 + 1 = 5. Even if it is the only node that replies, it is more than half of group1 (3 > 5 ACK). If it is the weight configuration at this time, group1 is also counted as a successful server.

It is also important to note that when ZK reads these configurations, the sum of the weights of each group is calculated. If the total weight of a group is 0, then the group is removed from the counting rule.

Unlike the default overhalf mechanism, it is possible for Observer to participate in the use of weight configuration.

3.2 recommended configuration

After talking for a long time, how to enable the counting rule of this weight?

(recommended) the configure dynamicConfigFile option in zoo.cfg is used to specify the path address of the dynamic configuration, moving all configurations starting with server, group, and weight to the configuration file of that path.

Configure all configurations at the beginning of server, group, and weight directly in the zoo.cfg file

As long as the configuration at the beginning of group or weight is found by ZK in the configuration file, the weighted vote counting rule is enabled, otherwise the default overhalf mechanism is used.

Our previous server prefix configuration is as follows:

Server.1=zoo1:2888:3888:participant

The format of the actual server configuration should be like this, and the client port can be configured in the server configuration (after the final semicolon)

Server.1=zoo1:2888:3888:participant;2181

If configured in this way, there is no need to configure the clientPort option in zoo.cfg.

So according to the recommended configuration, zoo.cfg will configure these (please adjust the path according to the reader's computer)

TickTime=2000initLimit=10syncLimit=5dataDir=/Users/junjiexun/develop/zk/zk01/datadynamicConfigFile=/Users/junjiexun/develop/zk/zk01/conf/zoo.dyn.cfg...

I can configure it in the / Users/junjiexun/develop/zk/zk01/conf/zoo.dyn.cfg file.

Server.10000000000=127.0.0.1:2888:3888:participant;2181server.2=127.0.0.1:2887:3887;2182server.3=127.0.0.1:2886:3886;2183group.1=10000000000:2:3weight.10000000000=1weight.2=1weight.3=1

For more details, you can check the official documentation.

3.3 dynamic modification

So the question is, after I have configured it in this way, how can I dynamically add or delete nodes to the cluster?

The client of Java provides a method of getConfig

ZooKeeper client = new ZooKeeper ("127.0.0.1 null 2181", 3000, null); byte [] config = client.getConfig (false, null); System.out.println (new String (config)); client.close ()

The printed result is

Server.2=127.0.0.1:2887:3887:participant;0.0.0.0:2182server.3=127.0.0.1:2886:3886:participant;0.0.0.0:2183server.10000000000=127.0.0.1:2888:3888:participant;0.0.0.0:2181group.1=2:3:10000000000weight.2=1weight.3=1weight.10000000000=1version=0

This information looks a bit like but a little different from the zoo.dyn.cfg we configured, the difference is actually the format that ZK helps us to complete automatically, and where is the returned data ZK stored? When ZK starts, it creates the following nodes in the root path by default

/ |-- zookeeper |-- config |-- quota

The data returned by getConfig is actually the data of / zookeeper/config node, and the permission of this node is only Read. If you try it with getData, the data returned will be the same.

ZooKeeper client = new ZooKeeper ("127.0.0.1 null 2181", 3000, null); byte [] config = client.getData ("/ zookeeper/config", false, null); System.out.println (new String (config)); client.close ()

Now that you can get this configuration, how do you modify it? ZK officially provides two ways: command line and Java API.

If you use Java's built-in command line tool, there is a reconfig command among the supported commands, with the following arguments:

Reconfig [- s] [- v version] [[- file path] | [- members serverID=host:port1:port2;port3 [,...] *]] | [- add serverId=host:port1:port2;port3 [,...]] * [- remove serverId [,...] *]

The other is to use the client code of Java. We have been using the ZooKeeper class before, and it also has a subclass called ZooKeeperAdmin, which has a reconfigure method that can modify the configuration. I'll demonstrate it below, but before I do that, I must explain the feature of dynamically modifying the configuration.

Dynamic configuration modification can be divided into incremental and non-incremental methods.

Because what is actually modified is the data of the / zookeeper/config node, and this node has only Read permissions by default, either use the administrator rights directly to modify the operation, or configure zookeeper.skipACL=yes in the environment variable to skip ACL verification

When modifying the configuration incrementally, the vote counting rules of the cluster must be more than half of the mechanism!

When you modify the configuration in a non-incremental manner, both mechanisms can be used.

Removing Follower from the cluster configuration is equivalent to downgrading it to Observer. It can still provide servers, and it can also accept messages from Leader.

Removing Leader from the cluster configuration will have a greater performance impact. The whole cluster cannot provide services until a new Leader is selected. Please try not to do so.

It is much easier to add more nodes, and the new nodes will automatically synchronize data with Leader.

3.3.1 incremental deletion of nodes

Suppose I have a total of three nodes, using more than half of the mechanism (it has to be). The three ID are 1000000000000, 2 and 3 respectively. We try to delete the node whose ID is 3. I use the direct configuration skipACL to skip permission verification (same below).

ZooKeeperAdmin client = new ZooKeeperAdmin ("127.0.0.1 null 2181", 3000, null); List leavingServers = new ArrayList (); leavingServers.add ("3"); byte [] reconfigure = client.reconfigure (null, leavingServers, null,-1, null); System.out.println (new String (reconfigure)); client.close ()

The data returned by this API is the configuration information after the modification of / zookeeper/config. You can see that the data related to 3 in the new configuration disappears.

Server.2=127.0.0.1:2887:3887:participant;0.0.0.0:2182server.10000000000=127.0.0.1:2888:3888:participant;0.0.0.0:2181version=400000004

What does this version=400000004 do? ZK also provides version control for modified configurations by default. After successful startup, a file will be automatically generated under your configured dynamicConfigFile path. My zoo.cfg.dynamic.300000000 readers here may be different from mine. This 300000000 is the version number, and when I delete the node with ID 3, ZK automatically generates a file zoo.cfg.dynamic.400000004. This 400000004 is the new version number. If we have a requirement for the version number of the current cluster configuration when we modify it, we can enter the target version number in the fourth parameter of the reconfigure method. In my example,-1 represents ignoring the version number. And the version field of delete and setData is the same purpose.

3.3.2 incrementally add nodes

Let's add node 3 back.

ZooKeeperAdmin client = new ZooKeeperAdmin ("127.0.0.1 null 2181", 3000, null); List joiningServers = new ArrayList (); joiningServers.add ("server.3=127.0.0.1:2886:3886;2183"); byte [] reconfigure = client.reconfigure (joiningServers, null, null,-1, null); System.out.println (new String (reconfigure)); client.close ()

The resulting new configuration is

Server.10000000000=127.0.0.1:2888:3888:participant;0.0.0.0:2181server.2=127.0.0.1:2887:3887:participant;0.0.0.0:2182server.3=127.0.0.1:2886:3886:participant;0.0.0.0:2183version=400000013

Node 3 has been added back, and the version number has changed again, and another zoo.cfg.dynamic.400000013 file has been added.

3.3.3 non-incremental ZooKeeperAdmin client = new ZooKeeperAdmin ("127.0.0.1 null 2181", 3000, null); List newMembers = new ArrayList (); newMembers.add ("server.10000000000=127.0.0.1:2888:3888:participant;2181"); newMembers.add ("server.3=127.0.0.1:2886:3886;2183"); byte [] reconfigure = client.reconfigure (null, null, newMembers,-1, null); System.out.println (new String (reconfigure)); client.close ()

This is equivalent to deleting the node with an ID of 2 (of course, you can also add a node, so I won't demonstrate it here)

3.4 Section

The three different ways actually correspond to the three parameters joiningServers, leavingServers, and newMembers of Java API, and the Java API parameters can also be achieved by using String (comma separated) in addition to List. Dynamic addition and deletion of service nodes allows us to adjust the service capacity of the entire ZK cluster without downtime (personally, I think dynamic addition is more useful).

The counting rule of grouping weight provides a new ticket return strategy, especially with dynamic configuration, the weight can be modified at run time, but generally speaking, the counting rule of grouping weight is more chicken. I don't know in what kind of scenarios (high-performance machines can have more weight? The problem is that now they are all cloud services, containerized and virtualized, and the configuration of the machine can be adjusted dynamically, and the general configuration of the machine is the same.

If readers have ideas on the use of grouping weights, you can share them with you. Oh, for more dynamic configuration, please refer to the official documentation.

IV. ZK monitoring

The new Metrics after ZK 3.6is a monitoring indicator that ZK provides to users. It is also said on the official website that it can be used in combination with Prometheus or Grafana. What? You haven't used it at all, you haven't even heard of it! Coincidentally, I have never played with these two things. I will take this opportunity to learn with you and build a Hello World, but because this series is based on ZK, everything is simple to configure, Let's GO!

4.1 Prometheus

Mac installation is very simple, other platforms can go to the official website to download compressed packages

$brew install prometheus

My default installation path here is / usr/local/Cellar/prometheus/2.23.0

Before that, you need to add two lines to the configuration zoo.cfg of the ZK node.

MetricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvidermetricsProvider.httpPort=7000

I have three nodes locally, and the other two nodes need to be changed to 7001 and 7002, because the ports cannot be duplicated.

Then modify the default configuration of Prometheus, and change the path to / usr/local/etc/prometheus.yml on my computer to

Global: scrape_interval: 15sscrape_configs:-job_name: "test-zk" static_configs:-targets: ["localhost:7000", "localhost:7001", "localhost:7002"]

Job_name can be started at will, focusing on targets destination address and scrape_interval access interval. After modification, you can start Prometheus.

$cd / usr/local/Cellar/prometheus/2.23.0 $. / bin/prometheus-- config.file=/usr/local/etc/prometheus.yml

Then visit localhost:9090 and you will see the following interface:

As long as you check Enable autocomplete, you can enter it in the input box, and you can get a prompt right away. I'll type in a few parameters here at random.

This is the end of the simple demonstration, and the rest will be left to the reader.

4.2 Grafana

Mac installation of Grafana is also very easy

$brew install grafana

After the installation is complete, you can start it with the command

$grafana-server-config=/usr/local/etc/grafana/grafana.ini-homepath / usr/local/share/grafana-packaging=brew cfg:default.paths.logs=/usr/local/var/log/grafana cfg:default.paths.data=/usr/local/var/lib/grafana cfg:default.paths.plugins=/usr/local/var/lib/grafana/plugins

The default port of Grafana is 3000, and the default username and password for accessing localhost:3000 is admin, so you can see the home page.

Where did the number 10465 come from? Templates for official documents

The great task has been completed! Much prettier than Prometheus ~

This is the end of the ZK surveillance. Traditional kung fu, so far.

Introduction of ZK Visualization Open Source Project

It is too troublesome to use the command line to operate ZK, so visualization is necessary. Here are some good visualization clients, some local clients and some Web services. Let's get them as needed.

PrettyZoo: https://github.com/vran-dev/PrettyZoo, visual GUI client, each platform has installation files. When you need to connect to the ZK server, it is very convenient to have such a tool on hand.

Zkdash: https://github.com/ireaderlab/zkdash, JavaScript + Python visual Web client, is a Web service that can be run directly. The disadvantage is that it is developed by Python2.7. If secondary development is not needed, then there is no problem.

Zoonavigator-web: https://github.com/elkozmon/zoonavigator-web, a visual Web client written by TypeScript, is a Web service that can be run directly

Visual-zookeeper: https://github.com/ghostg00/visual-zookeeper, client written by Electron + React

This is the end of "how to use four-word Command and Admin Server". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.