Shulou(Shulou.com)06/02 Report--

This article introduces the knowledge of "Why Java is not clear about memory troubleshooting". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

GC log configuration

But you are not always by the machine's side every time something goes wrong. Real-time performance cannot be guaranteed by manual operation. Therefore, it is strongly recommended that you make the GC log output in more detail, so that it will be more comfortable when something goes wrong.

In fact, this requirement seems to me to be mandatory.

Many students come up and say that my memory has overflowed. But you and it want some log information, a stack, and a snapshot saved on the spot. None of them. This is purely funny.

The following are the JDK8 or the following GC log parameters, which you can see is still very long.

#! / bin/sh LOG_DIR= "/ tmp/logs" JAVA_OPT_LOG= "- verbose:gc" JAVA_OPT_LOG= "${JAVA_OPT_LOG}-XX:+PrintGCDetails" JAVA_OPT_LOG= "${JAVA_OPT_LOG}-XX:+PrintGCDateStamps" JAVA_OPT_LOG= "${JAVA_OPT_LOG}-XX:+PrintGCApplicationStoppedTime" JAVA_OPT_LOG= "${JAVA_OPT_LOG}-XX:+PrintTenuringDistribution" JAVA_OPT_LOG= "${JAVA_OPT_LOG }-Xloggc:$ {LOG_DIR} / gc_%p.log "JAVA_OPT_OOM="-XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=$ {LOG_DIR}-XX:ErrorFile=$ {LOG_DIR} / hs_error_pid%p.log "JAVA_OPT=" ${JAVA_OPT_LOG} ${JAVA_OPT_OOM} "JAVA_OPT=" ${JAVA_OPT}-XX:-OmitStackTraceInFastThrow

The following is the log configuration for JDK9 and above. You can see that its configuration has all changed and is not backward compatible. The change made by Java still hurts.

#! / bin/sh LOG_DIR= "/ tmp/logs" JAVA_OPT_LOG= "- verbose:gc" JAVA_OPT_LOG= "${JAVA_OPT_LOG}-Xlog:gc,gc+ref=debug,gc+heap=debug,gc+age=trace:file=$ {LOG_DIR} / gc_%p.log:tags,uptime,time,level" JAVA_OPT_LOG= "${JAVA_OPT_LOG}-Xlog:safepoint:file=$ {LOG_DIR} / safepoint_%p.log:tags,uptime,time Level "JAVA_OPT_OOM="-XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=$ {LOG_DIR}-XX:ErrorFile=$ {LOG_DIR} / hs_error_pid%p.log "JAVA_OPT=" ${JAVA_OPT_LOG} ${JAVA_OPT_OOM} "JAVA_OPT=" ${JAVA_OPT}-XX:-OmitStackTraceInFastThrow "echo $JAVA_OPT

Once a problem is found, you can use the GC log to quickly locate the problem in the heap. But it's not for you to look at it line by line, it's too inefficient. Because the diary may be very long, and it may not be readable. At this time, you can use some online tools to help solve the problem. I often use gceasy, and here is a screenshot of it.

Http://gceasy.io

Not with the GC log, because it only records some changes in the heap space, as for some resource changes in the operating system, it is not known. So, if you have a monitoring system, you can also help when looking for problems. You can see some changes in system resources from the figure below.

Overflow exampl

Heap overflow

The code.

Journal.

Java-Xmx20m-Xmn4m-XX:+HeapDumpOnOutOfMemoryError-OOMTest [18.386s] [info] [gc] GC (10) Concurrent Mark 5.435ms [18.395s] [info] [gc] GC (12) Pause Full (Allocation Failure) 18m-> 18m (19m) 10.572ms [18.400s] [gc] GC (13) Pause Full (Allocation Failure) 18m-> 18m (19m) 5.348ms Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at OldOOM.main (OldOOM.java:20)

The reaction of jvisualvm.

Metaspace overflow

The code.

Journal.

Java-Xmx20m-Xmn4m-XX:+HeapDumpOnOutOfMemoryError-XX:MetaspaceSize=16M-XX:MaxMetaspaceSize=16M MetaspaceOOMTest 6.556s] [info] [gc] GC (30) Concurrent Cycle 46.668ms java.lang.OutOfMemoryError: Metaspace Dumping heap to / tmp/logs/java_pid36723.hprof.

The reaction of jvisualvm.

Direct memory overflow

The code.

Journal.

Java-XX:MaxDirectMemorySize=10M-Xmx10M OffHeapOOMTest Exception in thread "Thread-2" java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory (Bits.java:694) at java.nio.DirectByteBuffer. (DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect (ByteBuffer.java:311) at OffHeapOOMTest.oom (OffHeapOOMTest.java:27).

Stack overflow

The code.

Journal.

Java-Xss128K StackOverflowTest Exception in thread "main" java.lang.StackOverflowError at java.io.PrintStream.write (PrintStream.java:526) at java.io.PrintStream.print (PrintStream.java:597) at java.io.PrintStream.println (PrintStream.java:736) at StackOverflowTest.a (StackOverflowTest.java:5)

Which code is prone to problems?

Forgot to rewrite hashCode and equals

Look at the code below. Because the hashCode and equals methods of the Key class are not overridden. As a result, all objects put into the HashMap cannot be taken out. They're out of touch with the outside world.

The following article describes its principle in detail.

The result set is out of control

Don't think this code is funny. In the actual work of review, xjjdog has found this kind of painful code more than once. This may be due to a rush, or it may be that you have just learned to write Java. There is a good chance that this line of code will step into the hole.

The condition is out of control

The code. Similarly, the condition is out of control, when a condition is not satisfied, it will cause the result set to get out of control. You can take a look at the following code, what happens when fullname and other are empty?

Universal parameter

Some students use various Object and HashMap to exchange information. There is nothing wrong with this code when it is running properly, but if something goes wrong, it is almost impossible to troubleshoot. Troubleshooting parameters, troubleshooting stack, troubleshooting call chain, all invalidated.

Some preventive measures

Reduce the frequency of creating large objects: such as passing byte arrays

Don't cache too much heap data: use guava's weak reference schema

The scope of the query must be controllable, such as database and table middleware, ES, etc.

Be sure to drop close when you run out of resources: you can use the new try-with-resources syntax

Use less intern: if the string is too long and cannot be reused, it will cause memory leaks

Reasonable Session timeout

Use less third-party native code and use Java instead of

Reasonable pool size

XML (SAX/DOM), JSON parsing should pay attention to the object size

Case study one

This is the most common situation. Knowing this way, you can deal with most memory overflows and memory leaks.

Phenomenon

Environment: CentOS7,JDK1.8,SpringBoot

G1 garbage collector

There was no problem when it just started. After slowly increasing the volume, OOM occurred.

The system automatically generates heapdump files.

Temporary solution: restart, but the problem is still found

Information collection

Log: GC log information: memory suddenly increases and drops, and changes rapidly

Stack: Thread Dump files: most of them block on a method

Stress testing: using wrk for stress testing, 20 users are found to be concurrent and memory overflow

Wrk-T20-C20-d300s http://127.0.0.1:8084/api/test

MAT analysis

Stack file fetch:

Jmap-dump:format=b,file=heap.bin 37340 jhsdb jmap-binaryheap-pid 37340

The MAT tool is developed on the eclipse platform and is itself a Java program. Analyze the Heap Dump file: it is found that a large number of report objects are created in memory.

Through the menu Find Leaks, one click to find black Li Kui.

Just dig down according to the prompt.

Solve

Analysis results:

There is a large amount of data query service in the system, and merge in memory

When the concurrency reaches a certain degree, a large amount of data will be accumulated into memory for operation.

Solution:

Refactoring the query service to reduce the fields of the query

Use SQL queries instead of memory stitching to avoid operations on result sets

Example: find the intersection of two lists

Case study II

Phenomenon

Environment: CentOS7,JDK1.8,JBoss

CMS garbage collector

Operating system CPU resource exhaustion

Access to any interface, the response is very slow

Analysis.

It is found that GC works very well every time, but very frequently.

Learned that the in-heap cache is used and the capacity set is relatively large.

Cache filling is very fast!

Conclusion:

After a very large cache is opened, the GC fills up quickly, resulting in frequent GC

Case study three

Phenomenal java process exits abnormally

The java process simply disappears

No dump file is left.

The GC log is normal

When the monitoring found death, the memory in the heap was very small, and there was still a lot of space left in the heap.

Analysis.

XX:+HeapDumpOnOutOfMemoryError doesn't work.

Monitoring found that the operating system memory continued to increase.

All of the following situations will cause the program to exit without responding.

Dmesg oom-killer was killed by the operating system

System.exit ()

Java com.cn.AA & after the terminal shuts down

Kill-9

Solve

Found:

It was found in the dmesg command that it was indeed oom-kill.

Resolve:

Allocate less memory to JVM to make room for other processes

This is the end of the content of "Why Java is not clear memory troubleshooting". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.