Shulou(Shulou.com)06/03 Report--

This article mainly introduces "how to debug a large open source project ZooKeeper". In daily operation, I believe many people have doubts about how to debug a large open source project ZooKeeper. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts of "how to debug a large open source project ZooKeeper". Next, please follow the editor to study!

I. Source code debugging

It is better to teach people to fish than to teach people how to fish

I always believe that "the truth learned from paper is not profound". In the end, if readers want to really understand the internal principles of ZK, it is necessary to read the source code. If you have the ability of naked eye Debug as I do, you can actually not bother to build a debugging environment for source code.

But if not, it would be nice to download the ZK source code, run directly with the well-known IDE, and then directly break the points where you need to learn.

1.1 download source code

ZooKeeper 3.6.2 Source download Page

Choose any one with fast download speed in the above link, and click to download the compressed package. After the download is completed, you will get the following directory structure.

. ├── zookeeper-server ├── zookeeper-recipes ├── zookeeper-metrics-providers ├── zookeeper-jute ├── zookeeper-it ├── zookeeper-docs ├── zookeeper-contrib ├── zookeeper-compatibility-tests ├── zookeeper-client ├── zookeeper-assembly ├── zk-merge-pr.py ├── pom.xml ├── owaspSuppressions.xml ├── excludeFindBugsFilter.xml ├── dev ├── conf ├── checkstyleSuppressions.xml ├── checkstyle-strict.xml ├── checkstyle- Simple.xml ├── bin ├── README_packaging.md ├── README.md ├── NOTICE.txt ├── LICENSE.txt ├── Jenkinsfile-PreCommit └── Jenkinsfile

There is pom.xml in the directory, so ZK needs to compile the whole project through maven. Make sure your maven is installed first.

$mvn-- versionApache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00) Maven home: / your/maven/home/apache-maven-3.5.4Java version: 1.8.000181, vendor: Oracle Corporation, runtime: / Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jreDefault locale: zh_CN, platform encoding: UTF-8OS name: "mac os x", version: "10.16", arch:" x86deccf64 ", family:" mac "

If there is such an output indicating that maven is installed successfully, I will skip the specific installation process here. If you have any difficulties, you can leave us a message.

1.2 compile the project

Go to the same level directory as pom.xml and enter

$mvn install-DskipTests=true

You will see that the project is being compiled, and when the final output BUILD SUCCESS is output, the project compilation is complete.

[INFO] Reactor Summary: [INFO] [INFO] Apache ZooKeeper 3.6.2.. SUCCESS [3.621 s] [INFO] Apache ZooKeeper-Documentation. SUCCESS [2.086 s] [INFO] Apache ZooKeeper-Jute. SUCCESS [10.633 s] [INFO] Apache ZooKeeper-Server. SUCCESS [19.246 s] [INFO] Apache ZooKeeper-Metrics Providers. SUCCESS [0.108 s] [INFO] Apache ZooKeeper-Prometheus.io Metrics Provider. SUCCESS [1.286 s] [INFO] Apache ZooKeeper-Client. SUCCESS [0.083 s] [INFO] Apache ZooKeeper-Recipes. SUCCESS [0.092 s] [INFO] Apache ZooKeeper-Recipes-Election. SUCCESS [0.244 s] [INFO] Apache ZooKeeper-Recipes-Lock. SUCCESS [0.259 s] [INFO] Apache ZooKeeper-Recipes-Queue. SUCCESS [0.295 s] [INFO] Apache ZooKeeper-Assembly.... SUCCESS [5.425 s] [INFO] Apache ZooKeeper-Compatibility Tests. SUCCESS [0.072 s] [INFO] Apache ZooKeeper-Compatibility Tests-Curator 3.6.2 SUCCESS [0.432 s] [INFO]-[INFO] BUILD SUCCESS [INFO]- -[INFO] Total time: 44.263 s [INFO] Finished at: 2021-01-22T13:49:30+08:00 [INFO]- -1.3 Open and configure the project

Then you can open this directory through your IDE. I'm using idea here.

Then start configuring Run/Debug Configurations

Click + to add a new configuration

Select Application

1.3.1 standalone startup configuration

Then configure to fill in or select according to the following figure

Give this configuration a strong name first.

Select Modify options to open the submenu

Make sure that all three suboptions in the menu in the figure are selected (preceded by √)

Then let's look at the specific configuration.

The unzipped project path on my computer is / Users/junjiexun/Desktop/apache-zookeeper-3.6.2 readers, please modify it according to your own situation.

Choose your local jdk (I don't know if the other version is 1.8 locally, but the lower version certainly won't work, because some 1.8 words are used in the source code)

Select zookeeper

Configure VM options with the content of-Dlog4j.configuration=file:/Users/junjiexun/Desktop/apache-zookeeper-3.6.2/conf/log4j.properties. If it is not configured, the log cannot be output.

Specify the startup class org.apache.zookeeper.server.ZooKeeperServerMain

Command line parameters are required for stand-alone startup, which is 2181 / Users/junjiexun/Desktop/apache-zookeeper-3.6.2/data

This should be filled in automatically without modification. Anyway, the content is / Users/junjiexun/Desktop/apache-zookeeper-3.6.2.

Click the + in the middle to add the package path, which is org.apache.zookeeper.server.*

Then click Apply and OK to finish saving.

Then click on this bug to start it.

2021-01-22 15 main:NIOServerCnxnFactory@674 12 main:NIOServerCnxnFactory@674 16319 [myid:]-INFO [main:NIOServerCnxnFactory@674]-binding to port 0.0.0.0Univ 21812021-01-22 15 15 purl 16413 [myid:]-INFO [main:WatchManagerFactory@42]-Using org.apache.zookeeper.server.watch.WatchManager as watch manager2021-01-22 15 purl 12purl 16413 [myid:]-INFO [main:WatchManagerFactory@42]-Using org.apache.zookeeper .server.watch.watch.WatchManager as watch manager2021-01-22 15 INFO 12 main:ZKDatabase@132 16413 [myid:]-INFO [main:ZKDatabase@132]-zookeeper.snapshotSizeFactor = 0.332021-01-22 15 15 12 main:ZKDatabase@132 16413 [myid:]-INFO [main:ZKDatabase@152]-zookeeper.commitLogCount=5002021-01-22 15 15 14 12 main:SnapStream@61 16429 [myid:]-INFO [main:SnapStream@61]-zookeeper.snapshot.compression.method = CHECKED2021-01-22 15 15 Switzerland 1212 main:SnapStream@61 16432 [myid:] -INFO [main:FileSnap@85]-Reading snapshot / Users/junjiexun/Desktop/apache-zookeeper-3.6.2/data/version-2/snapshot.22021-01-22 15 Users/junjiexun/Desktop/apache-zookeeper-3.6.2/data/version-2/snapshot.22021 12 Reading snapshot 16444 [myid:]-INFO [main:DataTree@1737]-The digest value is empty in snapshot2021-01-22 15 15 14 12 Reading snapshot 16480 [myid:]-INFO [main:ZKDatabase@289]-Snapshot loaded in 67 ms Highest zxid is 0x2 Digest is 13719855042021-01-22 15 INFO 12 main:FileTxnSnapLog@470 16481 [myid:]-INFO [main:FileTxnSnapLog@470]-Snapshotting: 0x2 to / Users/junjiexun/Desktop/apache-zookeeper-3.6.2/data/version-2/snapshot.22021-01-22 15 15 12 ms2021 16488 [myid:]-INFO [main:ZooKeeperServer@529]-Snapshot taken in 6 ms2021-01-22 15 15 12 charge 16444 [myid:]-INFO [ProcessThread (sid:0 cport:2181):: PrepRequestProcessor @ 136]-PrepRequestProcessor (sid:0) started ReconfigEnabled=false2021-01-22 15 INFO 12 main:RequestThrottler@74 16546 [myid:]-INFO [main:RequestThrottler@74]-zookeeper.request_throttler.shutdownTimeout = 100002021-01-22 15 12 main:RequestThrottler@74 16623 [myid:]-INFO [main:ContainerManager@83]-Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=02021-01-22 15 15 INFO 12 main:ZKAuditProvider@42 16628 [myid:]-INFO [main:ZKAuditProvider@42]-ZooKeeper audit is disabled

See the log output, if there is no error, then it is a success!

Then we can test it with the client.

ZooKeeper client = new ZooKeeper ("127.0.0.1 null 2181", 3000, null); List children = client.getChildren ("/", false); System.out.println (children); client.close ()

Output as

[zookeeper]

The stand-alone version is done! Let's try the cluster version next.

1.3.2 Startup configuration of cluster version

Sometimes we need to debug the logic of the cluster version of ZK, but the previous stand-alone version is not enough, and I recommend decompressing the previous source code package to two different directories, and then opening the two directories through IDE to completely simulate two different nodes. The configuration of the cluster version is similar to that of the stand-alone version. Let's see what are the differences? I'm going to start the two nodes here with a myid of 1 and 2.

First copy the default zoo_sample.cfg and rename it to zoo.cfg, or you can rename it directly

Create a new data directory (if not) and create a new text file under it. The myid text content is 1.

Then edit the zoo.cfg:

# modify dataDir=/Users/junjiexun/Desktop/apache-zookeeper-3.6.2/data# and add the following two lines of server.1=127.0.0.1:2888:3888server.2=127.0.0.1:2887:3887

The specific configuration is as follows:

The startup class is different, and the cluster is org.apache.zookeeper.server.quorum.QuorumPeerMain.

The command line argument is different, the zoo.cfg path is passed in, and my path is / Users/junjiexun/Desktop/apache-zookeeper-3.6.2/conf/zoo.cfg

Then configure the second node, and I assume that the project directory of the second node is / Users/junjiexun/Desktop/apache-zookeeper-3.6.2-bak

The second node modifies the contents of the myid file to 2

The content in zoo.cfg is

# modify dataDir=/Users/junjiexun/Desktop/apache-zookeeper-3.6.2-bak/data#, because my two nodes are in the same machine, so the port cannot be duplicated. ClientPort=2182# also adds the following two lines of server.1=127.0.0.1:2888:3888server.2=127.0.0.1:2887:3887

The argument to the command line is / Users/junjiexun/Desktop/apache-zookeeper-3.6.2-bak/conf/zoo.cfg

Everything else I didn't mention is the same as Node 1.

Let's try to start two nodes.

2021-01-22 15 myid=1 44myid=1 08461 [myid:1]-INFO [myid=1] (plain= [0VOV0VOV0V0VOVO]: 2181) (secure=disabled): WatchManagerFactory@42]-Using org.apache.zookeeper.server.watch.WatchManager as watch manager2021-01-22 1544VOVA 08461 [myid:1]-INFO [QuorumPeer [myid=1] (plain= [0Vl0Vl0VV0]: 2181) (secure=disabled): WatchManagerFactory@42]-Using org.apache.zookeeper.server.watch.WatchManager as watch manager2021-01-22 15 INFO 44Ze44Ze08471 [myid:1]-INFO [QuorumPeer [myid=1] (plain= [0VOV0VOVO) 0VOVOVOVO]: 2181) (secure=disabled): Learner@677]-Learner received NEWLEADER message2021-01-22 15VOL44VOVO8471 [myid:1]-INFO [myid=1] (plain= [0VOVOVOVOVOVOULO): 0]: 2181) (secure=disabled): QuorumPeer@1811]-Dynamic reconfig is disabled We don't store the last seen config.2021-01-22 15 INFO 44Ze08471 [myid:1]-INFO [QuorumPeer [myid=1] (plain= [0VOV0VOVO) 0VOVOVO]: 2181) (secure=disabled): FileTxnSnapLog@470]-Snapshotting: 0x28100000001 to / Users/junjiexun/Desktop/apache-zookeeper-3.6.2/data/version-2/snapshot.281000000012021-01-22 15VR44VO8472 [myid:1]-INFO [QuorumPeer [myid=1] ( Plain= [0Snapshot taken in]: 2181) (secure=disabled): ZooKeeperServer@529]-Snapshot taken in 1 ms2021-01-22 15 Snapshot taken in 448525 [myid:1]-INFO [QuorumPeer [myid=1] (plain= [0VOVO): secure=disabled: Learner@661]-Learner received UPTODATE message2021-01-22 15Ranger 4408525 [myid:1]-INFO [QuorumPeer [myid=1] (plain=) [0Peer state changed]: 2181) (secure=disabled): QuorumPeer@868]-Peer state changed: following-synchronization2021-01-22 15 following 44bure44 8537 [myid:1]-INFO [QuorumPeer [myid=1] (plain= [0Peer state changed]: 2181) (secure=disabled): CommitProcessor@476]-Configuring CommitProcessor with readBatchSize-1 commitBatchSize 12021-01-22 15Suzhou 448537 [myid:1]-INFO [QuorumPeer [myid=1] (plain= [0CommitProcessor@438]: 2181) (secure=disabled): CommitProcessor@438]-Configuring CommitProcessor with 4 worker threads.2021-01-22 1515 14VOV 08544 [myid:1]-INFO [myid=1] (plain= [0VOV0VOV0]: 2181) (secure=disabled): RequestThrottler@74]-zookeeper.request_throttler.shutdownTimeout = 100002021-01-22 15Rd 448567 [myid:1] ]-INFO [QuorumPeer [myid=1] (plain= [0secure=disabled): QuorumPeer@863]-Peer state changed: following-broadcast

The final Peer state changed representative election is completed, and the node 1 posted is Follower, which is done!

After that, when you want to learn the process of source code, you can start the server directly. Isn't that enjoyable?

1.4 Source Code Reading Guide North

Server startup, cluster QuorumPeerMain#main, stand-alone ZooKeeperServerMain#main

Client ZooKeeper

Parsing configuration related, QuorumPeerConfig#parse

Memory Model (Little Red) DataTree

Callback notification (small yellow book) IWatchManager to view the implementation of the interface

Implement WatchManager by default

Optimization scheme WatchManagerOptimized

Electing FastLeaderElection#lookForLeader

Server instance, setting pipelined setupRequestProcessors method

Leader node LeaderZooKeeperServer

Follower node FollowerZooKeeperServer

Observer node ObserverZooKeeperServer

RequestProcessor of each assembly line staff to view the implementation of this interface

Persistent log FileTxnLog,snapshot FileSnap

Session Management SessionTrackerImpl#run

The protocol Record views the implementation of the interface.

1.5 Source Code Reading experience

Reading the source code of a large project must be a time-consuming and painstaking task. Here's what I've learned from reading the ZK source code:

Don't stick to the details! Large projects usually have a large amount of source code, and if you stare at every detail in the logic, you will be lost in the sea of source code.

Usually read the source code with a purpose. For example: how ZK converts protocols, how ZK is elected, and so on. After having the purpose, to see the relevant source code is to selectively ignore some other irrelevant details, you can use the method name or comments to have a perceptual understanding of the specific code block.

When you encounter a place that you don't understand, you can first go to the Internet to see if anyone has written a similar blog and stand on the shoulder of a giant.

In ZK, either indirectly or directly inheriting ZooKeeperThread is a thread object, and the main logic can look at the run method.

The important attribute of any class must be in the member field, and you can roughly infer the data structure behind the class by looking at the member field.

If there is a blocking queue field in the member property, the probability will be the embodiment of the producer-consumer model, and you can focus on the use of the blocking queue and when to put in and take out the element.

1.6 Summary

I used some pictures and texts to introduce how to debug the ZK source code locally, and how to read the source code scientifically. My local environment is Mac, and the IDE is idea. If your environment or tools are different from mine, if you encounter difficulties, you can also leave us a message.

2. Design patterns applied in ZK

ZK itself is a distributed application and an excellent open source project. Here I will briefly talk about the design patterns of applications in ZK that I have seen in reading the source code.

2.1 producers and consumers

This is a very representative application of the design pattern in ZK. ZK itself is the design of the Cmax S architecture. The request is sent by the client to the server, and the response is sent by the server to the client. Some functions of ZK are not accomplished by calling different methods in linear order, but are usually composed of producer threads, blocking queues and consumer threads. The producer thread puts some request objects received upstream into the blocking queue, and the current method returns, and then the consumer thread continuously fetches it from the blocking queue through a loop, and then completes the subsequent business logic. For example:

PrepRequestProcessor, the blocking queue is submittedRequests

SyncRequestProcessor, the blocking queue is queuedRequests

2.2 Factory Mod

There are implementations of some interfaces, and ZK itself provides default options, but if the consumer configures other implementations in the configuration, the ZK factory automatically creates those other implementations. For example:

When creating a ClientCnxnSocket, the client's IO implementation is selected according to the configuration of the zookeeper.clientCnxnSocket

When creating an IWatchManager, the server-side watch management implementation is selected according to the configuration of the zookeeper.watchManagerName.

When creating a ServerCnxnFactory, the server-side IO factory implementation is selected according to the configuration of the zookeeper.serverCnxnFactory

2.3 chain of responsibility model

As we have learned before, ZK server business logic processing is achieved by stringing XxxProcessor together. Processor does not care about the order of calls, but only through nextProcessor association. Different server roles can also greatly reuse code in this way.

In stand-alone mode: PrepRequestProcessor-> SyncRequestProcessor-> FinalRequestProcessor

Leader: LeaderRequestProcessor-> PrepRequestProcessor-> ProposalRequestProcessor-> CommitProcessor-> Leader.ToBeAppliedRequestProcessor-> FinalRequestProcessor in cluster mode

Follower: FollowerRequestProcessor-> CommitProcessor-> FinalRequestProcessor in cluster mode

Observer: ObserverRequestProcessor-> CommitProcessor-> FinalRequestProcessor in cluster mode

2.4 Policy Model

Zookeeper.snapshot.compression.method can be configured with different snapshot compression algorithms. When a snapshot file needs to be generated, it will be executed according to different compression algorithms:

Gz:GZIPInputStream

Snappy:SnappyInputStream

Default: BufferedInputStream

2.5 decorator mode

It is still the compression algorithm just now. What is provided is the unified processing object of CheckedInputStream. The above three compression implementations are packaged with CheckedInputStream. These objects are all subclasses of InputStream.

Switch (depending on configuration) {/ / embodiment of policy mode case GZIP: is = new GZIPInputStream (fis); break; case SNAPPY: is = new SnappyInputStream (fis); break; case CHECKED: default: is = new BufferedInputStream (fis);} / / are wrapped into return new CheckedInputStream (is, new Adler32 ()) of CheckedInputStream// decorator pattern At this point, the study on "how to debug a large open source project ZooKeeper" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.