Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "JVM online troubleshooting methods". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

CPU Biao Gao-Please use the linux operation and maintenance monitoring command

The following is an excerpt:

Idea: first find the Java process whose CPU is high, because your server will have multiple JVM processes. Then find the "problem thread" in that process, and finally find the problem code based on the thread stack information. Finally, the code is checked.

How to operate it?

Use the top command to find the process that consumes the most CPU, and remember the process ID.

Once again, use top-Hp [process ID] to find the thread with the highest CPU consumption, ID, and remember thread ID.

Through the jstack tool dump thread stack information provided by JDK to the specified file. Specific command: jstack-l [process ID] > jstack.log.

Since the thread ID is decimal and the thread ID in the stack information is hexadecimal, we need to convert the decimal to hexadecimal and use this thread ID to look in the stack. Using printf "% x\ n" [decimal number], you can convert decimal to hexadecimal.

Find the corresponding thread stack from the stack information through the hexadecimal number you just converted. You can see the clue from the stack.

From the experience of the landlord, there is generally no exit for a business dead cycle, and this situation can be repaired according to the business. And the C2 compiler will also preempt the CPU when performing the compilation. What is the C2 compiler? When a piece of code in Java is executed more than 10000 times (the default), it will be changed from interpretive execution to compiled execution, that is, compiled into machine code to increase speed. And that's what this C2 compiler does. How to solve the problem? After the project is online, it can be preheated by the stress testing tool, so that when the user actually accesses it, the C2 compiler will not interfere with the application. If it is caused by the GC thread, then it is most likely Full GC, so you need to optimize the GC.

Memory troubleshooting-Please use the JVM performance monitoring tool

The following is an excerpt:

After talking about the troubleshooting of CPU, let's talk about the troubleshooting of memory. Usually, the problem of memory is the problem of GC, because the memory of Java is managed by GC. There are two cases, one is a memory overflow, and the other is that there is no memory overflow, but the GC is not healthy.

Memory overflows can be done by adding the-XX:+HeapDumpOnOutOfMemoryError parameter, which outputs the dump file when the program's memory is overflowed.

With the dump file, you can analyze it through dump analysis tools, such as the commonly used MAT,Jprofile,jvisualvm and other tools, which can see where the overflow is, where a large number of objects have been created, and so on.

The second situation is more complicated. GC's health problems.

What is the usual state of a healthy GC? According to the experience of the landlord, YGC 5 seconds or so, each time no more than 50 milliseconds, FGC had better not, CMS GC once a day or so.

The optimization of GC has two dimensions, one is frequency, the other is time.

When we look at YGC, we first look at the frequency. If the YGC is more than 5 seconds, or even longer, it means that the system memory is too large and the capacity should be reduced. If the frequency is very high, it means that the Eden area is too small and the Eden area can be increased. But the capacity of the whole Cenozoic generation should be between 30-40% of the heap, and the ratio of eden,from to to should be around 8:1:1, which can be adjusted according to the size of the object's promotion.

What if the YGC takes too long? YGC has two processes, one is scanning, the other is replication, usually the scanning speed is very fast, but the replication speed is slower. If there are a large number of objects to copy each time, the STW time will be extended. Another case is StringTable. This data structure stores references to the continuous pool returned by the String.intern method. YGC scans this data structure (HashTable) every time, if the data structure is large. And without FGC, it will also lengthen the length of STW. Another situation is the virtual memory of the operating system. When the operating system happens to be swapping memory, it will also lengthen the length of STW.

Let's take a look at FGC. In fact, in FGC, we can only optimize the frequency, not the duration, because the duration is uncontrollable. How to optimize the frequency?

First of all, there are several reasons for FGC, 1 is insufficient memory in Old area, 2 is insufficient memory in metadata area, 3 is System.gc (), 4 is jmap or jcmd,5 is CMS Promotion failed or concurrent mode failure,6 JVM is pessimistic strategy that Old area cannot accommodate promotion objects after this YGC, so cancel YGC and advance FGC.

The usual optimization point is that insufficient memory in the Old area leads to FGC. If there are a large number of objects after FGC, it means that the Old area is too small, and the Old area should be expanded. If the effect after FGC is very good, it means that there are a large number of short-lived objects in the Old area. The point of optimization should be to let these objects be dropped by YGC in the new generation. The usual way is to increase the new generation. If there are large and short-lived objects, set the size of the object by parameters, do not let these objects enter the Old area, and check whether the promotion age is too young. If after YGC, a large number of objects are promoted ahead of time because they are unable to enter the Survivor area, the Survivor area should be increased, but not too large.

All of the above are optimization ideas, and we also need some tools to know the state of GC.

JDK provides a lot of tools, such as jmap, jcmd, etc. Oracle officially recommends using jcmd instead of jmap, because jcmd can really replace many functions of jmap. Jmap can print the distribution information of objects and dump files. Note that FGC will be triggered when jmap and jcmd dump files are used, and pay attention to the scene when using it.

Another common tool is jstat, which can view the details of GC, such as memory usage in areas such as eden, from,to,old, and so on.

Another tool is jinfo, which can see which parameters are used by the current jvm, and can also modify parameters without downtime.

Including some of the visualization tools we mentioned above to analyze dump files, MAT,Jprofile,jvisualvm, etc., these tools can analyze the files under jmap dump to see which object uses more memory, usually can find out the problem.

It is also very important to bring GC logs with you in the online environment!

This is the end of the content of "JVM online troubleshooting methods". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.