Shulou(Shulou.com)06/03 Report--

How to solve Rocketmq downtime, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

one

The time goes back to one night in December 2018, when I was getting ready to go online and went home after a demand. As soon as I clicked the release button, the alarm sounded. Damn it, can't I go home? Looking at the number of errors reported only one or two, it is concluded that it is only occasional, steady do not panic.

After sending out the rest of the machine, several of the same errors occurred. As a good (salty) show (fish) programmer, this problem must be tracked down to the end.

two

Skillfully query the error log

Org.apache.ibatis.exceptions.PersistenceException: # Error querying database. Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed

Looking at the abnormal information, I was lost in thought.

Ostensibly, the error report is due to the use of a closed data source.

When will the data source be shut down? Only when the process is killed

Is it because the application is not smooth enough when it is closed? The traffic will be removed first when it is released, so it should not go so far.

It was very late, aimlessly dragging the log, wearily looking for new clues, suddenly a word in the wrong log was introduced: "rocketmq"

Energetic, probably know why, there is a conscientious rocketmq consumer in the app has been consuming messages, when the app closed, external traffic was removed, but no one informed rocketmq consumer, so it threw an exception.

three

Out of my unprofound or even superficial understanding of rocketmq, its consumption uses the ack way. If an error is reported, the message will be retried later without losing the message, and if the consumer code is idempotent, there will be no business exception, anyway, it doesn't matter, because it's not the code I wrote.

After a glance at the consumer code (no code here, but you won't read it anyway), consumer registers a ShutdownHook,ShutdownHook in which consumer executes shutdown to exit gracefully, and sets the highest priority for the shutdownThread, but in practice, the highest priority of this thread is useless.

And we also know from the article "the principles of ShutdownHook" that ShutdownHook is executed concurrently, and spring container closure is also a ShutdownHook, which is not in order before.

After understanding the reason, I immediately thought of a solution similar to dubbo, wrote a graceful interface to turn off rocketmq cosnumer, and called it before applying the kill of the shutdown script, so as to solve the problem perfectly and hurry home from work, otherwise I would die suddenly.

four

When I fell asleep at night, I dreamt that my boss asked me to change all the systems, which scared me smart.

So I rethought the question the next day, and always felt that it was very inelegant to implement an interface in the application and call it in the stop script, and more importantly, it couldn't be copied to other projects, and I was lost in thought.

Since it is the order in which bean is destroyed when the spring container is closed, can we use spring's depend-on to straighten out the order? Just do it.

At first I encountered a dependency like this:

Hand-in-hand matching depend-on relationships in every bean of xml seems to work, too.

But when I open the second project, the dependency between its bean is roughly as follows:

Boy, 26 letters were almost not enough. I was in this mood at that time.

So I think at the current rate, all the projects may be 9102 years old.

five

After a while, I suddenly saw the official spring-boot-starter implementation of Rocketmq on the github dating website, so I clicked in to see its implementation. Boy, call 666 after reading it.

The official starter implements spring's SmartLifecycle interface, its start method can be called after all bean initialization, and the stop method will be called before the bean is destroyed, which is perfect for rocketmq consumer.

By the way, I also reviewed the closing of the spring container, and the code is in AbstractApplicationContext's doClose method. Here I summarize it into a picture:

As you can see from the figure above, before destroying the bean, the official starter chose to implement the LifeCycle API for closing the lifecycle bean and sending the LifeCycle.

six

It's time for me to report to the boss here. The reason why the rocketmq consumer release is not smooth is our posture. Although it has no impact on the business, it is not elegant. There are two solutions, boss, you can choose:

All changed to official starter, relying on spring-boot, official maintenance, high cost of renovation

Monitor ContextClosedEvent to achieve elegant shutdown. This piece can be encapsulated to allow the business side to introduce dependencies.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.