Shulou(Shulou.com)06/02 Report--

In this issue, the editor will bring you about why there is no zookeeper node in the ZooKeeper registry. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Overview

Specific reason: due to ZooKeeper data inconsistency

Repair operation: modify the ZooKeeper configuration as follows: delete all ZooKeeper data, restart ZooKeeper, and restart all dubbo providers/consumers services.

After a period of time, such problems did not occur.

The ZooKeeper configuration file is modified to:

ClientPort=2181dataDir=/data/zk-datadataLogDir=/data/zk-logstickTime=5000initLimit=10syncLimit=20maxClientCnxns=60server.0=zk-0:2888:3888server.1=zk-1:2888:3888server.2=zk-2:2888:3888 operating environment:

Dubbo version number: 2.5.3

ZooKeeper version number: 3.4.11

Im-service is the dubbo provider (provider)

Im-web is the dubbo consumer (consumer)

Problem scenario:

Some of the dubbo API calls failed. Check the call log of the explorer:

Generally speaking, this kind of Forbid consumer problem is to check whether there is a provider under the corresponding / providers node in the ZooKeeper registry. If there is no provider, it is to see if the provider starts normally.

So I looked at the children of the / dubbo/com.newbanker.im.service.WbsExpertService/providers node on the zk-0 node:

Ls / dubbo/com.newbanker.im.service.WbsExpertService/providers

It was found that the providers child node does not exist. In other words, the ZooKeeper registry thinks that dubbo providers is not starting properly, so take a look at the running status of dubbo providers:

Figure: dubbo provider is operating normally. I was confused at this point: docker thought the provider's im-service project was working properly, but the dubbo registry ZooKeeper thought the provider was not registered. There must be some misunderstanding.

Solution:

Restart the im-service service

After rebooting, everything ok!

Guess the reason:

After the Dubbo project was started, there was no provider. The project startup log is normal and the DUBBO service startup is not registered with zookeeper.

TickTime revised to 5000: 2019-09-17: no such problems have been found so far.

Extension problem

Similar problems have been found today, but there is one difference:

The child node of / dubbo/com.newbanker.im.service.WbsExpertService/providers in the ZooKeeper zk-0 node does not exist.

The child node of / dubbo/com.newbanker.im.service.WbsExpertService/providers in the ZooKeeper zk-1 node does not exist.

A child node of / dubbo/com.newbanker.im.service.WbsExpertService/providers exists in the ZooKeeper zk-2 node.

Faced with this kind of data inconsistency, I initially guessed that data synchronization was being done within the ZooKeeper cluster, resulting in data inconsistencies.

So use the. / bin/zkService.sh status command to check the status of each ZooKeeper node and find that everything is normal: the zk-2 node is leader,zk-0 and the zk-1 node is follower. That is, you encounter data inconsistencies across nodes in a ZooKeeper cluster. This does not match the data consistency we talked about at the beginning.

With the mentality of giving it a try, Baidu found:

Location process of inconsistent ZooKeeper data (3.4.11)

This blog post says that bug exists in ZooKeeper 3.4.11, and this bug is already fixed in ZooKeeper 3.4.12.

So I exported the ZooKeeper log and looked at it:

Why is there such a coincidence in the world? The ZooKeeper we use is really 3.4.11.

Solution:

That's right, restart the im-service service! (~ "▽") ~

Since the scope of impact of replacing the ZooKeeper version is unknown, we did not replace the ZooKeeper version to 3.4.12 for the time being, but restarted the dubbo provider project. After the restart, the ZooKeeper data is consistent and the dubbo consumer calling interface is normal.

Various reasons for this bug will be followed up later.

Think about:

The blog description in the guessing reason above is due to the fact that tickTime time is too short.

However, according to the extension problem above, the root cause is still ZooKeeper, that is to say, whether tickTime is too small or not, ZooKeeper data can not be inconsistent.

The focus now is to find out the cause of the ZooKeeper data inconsistency and resolve it!

Another bug with inconsistent ZooKeeper data:

(below: abnormal MarketingActivityService consumption)

(below: ZooKeeper zk-0 data)

(below: ZooKeeper zk-1 data)

(below: ZooKeeper zk-2 data)

In order to prove that there is no problem with the above figure, I also asked my operation and maintenance colleagues to enter the docker pod and view the ZooKeeper data:

ZooKeeper zk-0 data: (there is a dubbo:// indicating that there is a provider)

Ls / dubbo/com.newbanker.ac.service.MarketingActivityService/ providers [dubbo: / / 10.233.81.217 Phantom 23880pov. Newbanker.ac.service.MarketingActivityServiceAc.service.MarketingActivityServiceProduction1000 updated anyhostworthy trueworthy applicationapplications acompany services buffer8192 charting charsetsUTFave8clients nettypewriters default.delaymakesfault.service.filtermakers IdReceiveFilterfauldefault.timeout60000floor delaycards. Filtermakers IdReceiveFilterfauldefault.timeouts60000 delayhands to buy 1cards dubbobbots 2.5.3examples interfaceframes com.newbanker.ac.service.MarketingActivityServicereadreadsrequests 9methodological copies, previewbooks, previews.previewview, UpdateCostBameNameunreleaseumappwwnnnnappwappwfofoveduveduvendens, the originators, the originals, the catalogues, the originals, the LabelCount,updateChannel,queryAppPage,typeCount,selectByNos,add,previewQrcode,timingRelease,statusCount,cusView,list,untop,updateEvaluationPeriod,queryPage&payload=8388608&pid=8&queues=0&revision=0.0.1-SNAPSHOT&serialization=hessian2&server=netty&side=provider&threadpool=fixed&threads=100×tamp=1567157143813&version=1.0]

ZooKeeper zk-1 data: (no provider)

Ls / dubbo/com.newbanker.ac.service.MarketingActivityService/providers []

ZooKeeper zk-2 data: (there are two dubbo:// indicating two provider)

Ls / dubbo/com.newbanker.ac.service.MarketingActivityService/ providers [dubbo: / / 10.233.112.1919 freed 23880appert com.newbanker.ac.service.MarketingActivityServicehands clients. 1000 examples anyhostworthy trueworthy applications acclaves serviceholders 8192 chartered charters UTFmur8 examples clientbooks nettypecodes default.delaycolors words 1 examples default.perfect frames default.service.filtermakers IdReceiveFilterfilters fault.timeouts60000 delaycards, 1cards dubbow2.5.3examples interfacepieces com.newbanker.ac.service.MarketingActivityServicereads9 methods _ originviewresult Upcostdatebook _ LabelCount,updateChannel,queryAppPage,typeCount,selectByNos,add,previewQrcode,timingRelease,statusCount,cusView,list,untop,updateEvaluationPeriod,queryPage&payload=8388608&pid=8&queues=0&revision=0.0.1-SNAPSHOT&serialization=hessian2&server=netty&side=provider&threadpool=fixed&threads=100×tamp=1567391428907&version=1.0 Dubbo://10.233.81.217:23880/com.newbanker.ac.service.MarketingActivityService?accepts=1000&anyhost=true&application=ac-service&buffer=8192&charset=UTF-8&client=netty&default.delay=-1&default.retries=0&default.service.filter=entIdReceiveFilter&default.timeout=60000&delay=-1&dubbo=2.5.3&interface=com.newbanker.ac.service.MarketingActivityService&iothreads=9&methods=preview,costUpdate,release,selectByName,unrelease,appView,unrecommend,update,recommend,delete,timingDateUpdate,selectByNo,exportMarketingActivity,timingDateAdd,view,top,timingDateDelete,labelCount,updateChannel,queryAppPage,typeCount,selectByNos,add,previewQrcode,timingRelease StatusCount,cusView,list,untop,updateEvaluationPeriod,queryPage&payload=8388608&pid=8&queues=0&revision=0.0.1-SNAPSHOT&serialization=hessian2&server=netty&side=provider&threadpool=fixed&threads=100×tamp=1567157143813&version=1.0]

This tm is very embarrassing. In the same zk cluster, the data of each node is not consistent?!

Take a look at the ZooKeeper configuration of this environment: (according to the OPS, the configuration of ZooKeeper is killed by docker image, so there will be no problem with the configuration of ZooKeeper.)

/ zookeeper-3.4.11 # cat / conf/zoo.cfg clientPort=2181dataDir=/data/zk-datadataLogDir=/data/zk-logstickTime=2000initLimit=5syncLimit=2maxClientCnxns=60server.0=zk-0:2888:3888server.1=zk-1:2888:3888server.2=zk-2:2888:3888

Still don't have a clue.

-

2019-09-03 record:

Modify the ZooKeeper configuration file to:

ClientPort=2181dataDir=/data/zk-datadataLogDir=/data/zk-logstickTime=5000initLimit=10syncLimit=20maxClientCnxns=60server.0=zk-0:2888:3888server.1=zk-1:2888:3888server.2=zk-2:2888:3888

After modifying the configuration, delete the transaction log and snapshot files, restart the ZooKeeper cluster, and restart all dubbo services.

This is why there is no zookeeper node in the ZooKeeper registry shared by the editor. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.