Shulou(Shulou.com)06/02 Report--

Spring Boot false death diagnosis example analysis, in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

In the past two days, there is a problem of fake death of the service. The specific phenomenon is that the service no longer receives any requests, and the client will throw a Broken Pipe.

Check system statu

Execute top and find that both CPU and memory footprint are not high, but through the command

Netstat-n | awk'/ ^ tcp/ {+ + S [$NF]} END {for (an in S) print a, S [a]}'

It is found that a large number of CLOSE_WAIT ports are occupied, and the api of the service continues to be called. After waiting for the timeout, it is found that the number of CLOSE_WAIT has not increased, that is, the service is almost completely dead.

Check JVM condition

Suspected that the thread may have a deadlock, decided to first dump the thread situation, execute

Jstack > / tmp/thread.hump

It is found that the tomcat thread is basically normal and is in a parking state.

This is a bit strange, continue to wonder if GC caused STW, use jstat to check garbage collection

App@server:/tmp$ jstat-gcutil 1 2000 10 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 27.79 65.01 15.30 94.75 92.23 1338 44.375 1881 475.064 519.439

I was startled to see that the number of FGC actually exceeded that of YGC, with a duration of 475s. There must be something that triggered FGC, but fortunately we opened GC log.

It was found that Full GC caused by Allocation Failure occurred frequently in a period of time. And the use of the eden area is also very large, considering that there are frequent new objects escape to the old times caused problems. Asked the development of the business to confirm that there is an external docking API without paging, and a large number of objects may be generated after the query.

Since the external API is temporarily unable to contact the other party for modification, in order to solve the problem first, the original MaxNewSize is expanded, from 192MB to double. After several days of observation, it is found that gc basically tends to be normal.

S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 3.37 60.55 8.60 95.08 92.98 87 2.421 0 0.000 2.421

Dump heap before capacity expansion.

Jmap-dump:format=b,file=heapDump

When analyzing memory leaks through MAT, it is suspected to be a class in jdbc, but the overall heap capacity is not much.

The number of threads is analyzed, which is about 240, which is not very different from normal. And a large number of timed threads are in sleep.

In fact, this survey did not find the real reason, the indirect appearance is that FGC frequently leads to service fake death. And the acturator port is working properly, which causes the health check process to mistakenly think that the service is normal and does not trigger an alarm. If you encounter a similar situation, you are welcome to discuss it.

This is the answer to the case analysis of Spring Boot false death diagnosis. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.