Shulou(Shulou.com)06/03 Report--

This article to share with you is about how to troubleshoot the Dubbo interface duplicate cancellation problem, Xiaobian think it is very practical, so share it for everyone to learn, I hope you can gain something after reading this article, not much to say, follow Xiaobian to have a look.

I am responsible for the dubbo registration center self-developed in the company, and I often receive feedback from the business party that the dubbo interface is cancelled and an error is reported. After investigation, it is determined that the same interface calls the logout interface twice. Since our registry logout interface cannot be called repeatedly, the second call will report an error that the instance cannot be found because the instance has been logged out.

Although this error only prints an error log and does not affect the business, in the spirit of follow through, I decided to find out what happened, not to mention that repeated logouts also increase the end time of the application and affect the release rollback speed.

problem recurrence

Get the dubbo version of the business side, a version based on the internal customization of open source 2.7.3. The modification of this version mainly involves security vulnerability repair and some business adaptation. Write a demo to run, and then kill, and find that it is wrong.

To make sure it wasn't an internal fix, I tested it again with the open source version 2.7.3 and found that it still reported an error.

Also to make sure this is a bug, I changed dubbo version to 2.7.7 for testing and found that this version no longer reports errors.

Explains duplicate logouts at least a bug in open source dubbo 2.7.3 that has been fixed in later versions 2.7.7.

So there is a solution: upgrade dubbo, but if it were that simple there would be no article.

The internal dubbo has been modified. If you want to upgrade, you have to merge the changes to the new version. It is more difficult.

Even if the internal dubbo version is upgraded, it is impossible to push the business side to upgrade so quickly.

So you should first find out where the bug is causing it, then see if the registry extension can fix the problem, and if it can't fix it, you can only fix it in the internal dubbo version.

problem investigation

Suspicion ShutdownHook

As a result of studying ShutdownHook these days (click to view the original jump to "ShutdownHook Principle"), the first time suspected that ShutdownHook may have problems.

dubbo 2.7.3 code related to ShutdownHook implementation in the Dubbo ShutdownHook class, along with the code to sort out the following relationship

See dubbo itself and spring are registered ShutdownHook, more suspicious here is not ShutdownHook registration duplicate. So debug to see if the registration is repeated, here is a small experience, IntelliIDEA debugging ShutdownHook execution, manually kill the process will trigger debug, click the IDE on the close button will not trigger

In DubboShutdownHook.doDestroy hit a breakpoint, debug found that it will only be executed once, which means that ShutdownHook of spring and dubbo will only be registered once, how is this achieved? After many tests, I found dubbo a very cool design.

DubboShutdownHook has register and unregister methods, register and unregister ShutdownHook respectively, break points on both methods, and find such an interesting execution sequence at program startup:

To sum up, dubbo itself registered ShutdownHook, but if the Spring framework is used, the Spring framework cancels the ShutdownHook registered by dubbo during initialization, so that only the ShutdownHook of Spring is retained. The implementation of the code is only a few short lines

public static void addApplicationContext(ApplicationContext context) { CONTEXTS.add(context); if (context instanceof ConfigurableApplicationContext) { ((ConfigurableApplicationContext) context).registerShutdownHook(); DubboShutdownHook.getDubboShutdownHook().unregister(); } BeanFactoryUtils.addApplicationListener(context, SHUTDOWN_HOOK_LISTENER); }

So the suspected ShutdownHook problem proved to be no problem at all.

Continue troubleshooting from logout stack

Stable recurrence of the problem must be very good troubleshooting, with the help of IDE debug to see two logouts of the call stack, in the registry extension of the unregister method add breakpoint, you can see the following two different sources of stack information

The code reflects

That is, one ShutdownHook execution triggers two logouts.

Next, it's easier to investigate, debug step by step, explain it here

AbstractRegistryFactory. destrucyAll () is to destroy all registries, and when destroyed, it will investigate the registry's cancellation interface.

destroyProtocols is to destroy all protocols. The protocol of the registry gets the registry when destroying, and then calls the registry logout interface.

So how does dubbo 2.7.7 avoid this problem?

In dubbo 2.7.7 code, registry protocol gets registry slightly more dot code on destruction

Originally, after the registry was destroyed, the destroyed variable was set to true, so that when the registry protocol obtained the registry again, the original registry could not be obtained. What was obtained was an empty registry. Calling logout naturally had no effect.

Go back to github, this time PR is

https://github.com/apache/dubbo/pull/5450

This fix was fixed in 2.7.5

summary

dubbo duplicate logout problem exists in versions 2.7.0 ~ 2.7.4, fixed in 2.7.5, zk registry does not report error, may not be aware, but it does exist, it will also slow down the application shutdown speed

The problem can be solved in the registry extension, so that the destroy of registry can only be called once.

No matter how small the problem, free time to delve into it, you will receive some new knowledge, such as this dubbo ShutdownHook such clever design

The above is how to troubleshoot the Dubbo interface duplicate cancellation problem. Xiaobian believes that some knowledge points may be seen or used in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.