Shulou(Shulou.com)06/03 Report--

This article mainly explains "how to troubleshoot the problem of excessive memory in JVM". The content of the explanation in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to troubleshoot the problem of excessive memory in JVM".

There are generally two cases of excessive memory: memory overflow and memory leak.

Memory overflow: the memory allocated by the program exceeds the memory size of the physical machine, resulting in unable to continue to allocate memory, resulting in an OOM error

Memory leaks: objects that are no longer in use always occupy memory and are not released, resulting in this memory being wasted. Over time, objects with memory leaks accumulate, which will also cause physical machines to run out of memory and cause OOM errors.

Concrete operation

How to monitor JVM, we can learn about some actual operations through a case. You can see the following code. The following code only simulates one of the scenarios, a risk control scenario. Generally, when a bank or a third-party company issues a loan to a person, it will call that person's credit repayment ability and give the response rating.

Import java.math.BigDecimal;import java.util.ArrayList;import java.util.Date;import java.util.List;import java.util.concurrent.ScheduledThreadPoolExecutor;import java.util.concurrent.ThreadPoolExecutor;import java.util.concurrent.TimeUnit;public class FullGCTest {/ / simulated bank card class private static class CardInfo {/ / Bank card information records of small farmers BigDecimal price = new BigDecimal (10000000.0); String name = "herdsmen and small farmers"; int age = 18 Date birthdate = new Date (); public void m () {}} / / timed thread pool / / 50 thread pools, then set reject policy private static ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor (50, new ThreadPoolExecutor.DiscardOldestPolicy ()); public static void main (String [] args) throws Exception {executor.setMaximumPoolSize (50); for (;;) {modelFit () Thread.sleep;}} / * risk assessment for bank cards * / private static void modelFit () {List taskList = getAllCardInfo () / take out each message and taskList.forEach (info-> {/ / do something executor.scheduleWithFixedDelay (()-> {/ / call M method info.m ();}, 2,3, TimeUnit.SECONDS);} private static List getAllCardInfo () {List taskList = new ArrayList () / / for (int I = 0; I) for each query of 100 cards.

< 100; i++) { CardInfo ci = new CardInfo(); taskList.add(ci); } return taskList; }} 程序的设计其实比较简单，就是我们用信用卡的案例来进行说明，比如CardInfo就是信用卡类，我们把这个人对应的信用卡的记录都调用出来，之后做一些自己响应的业务处理方法来对它进行处理和计算，来看看我们这个模型是否符合modelFit，具体怎么做呢，在应用程序中有一个类是CardInfo，有一个方法叫做getAllCardInfo，每次都是拿100个出来，拿100个之后用线程池做计算，线程池用的是ScheduledThreadPoolExecutor (定时任务)，new出来线程池之后，50个线程池，然后做对应的业务逻辑处理，会调用 modelFit()，使用100毫秒模拟业务的停顿。 首先我们需要使用javac 命令将Java文件进行编译 javac FullGCTest.java进行编译，然后打印GC日志，进行风险监控 打印GC日志：java -Xms200M -Xmx200M -XX:+PrintGC FullGCTest

How do I know that JVM memory is too high?

In the company, if the JVM memory is too high, the operation and maintenance team usually receives the alarm message first, and then notifies the corresponding developer to check it, so how should the developer check it, or how to troubleshoot it?

1. Top to view the progress

After receiving the alarm message, take the top command to inquire.

[root@root ~] # top

Check out the growing memory and high CPU occupancy. After top you will see its PID (31061). It accounts for a relatively high proportion.

2. Top-Hp view thread

Find the process PID with high CPU usage. Here we use the command top-Hp 31061 to take the java process as an example. Will it list all the threads in this process? these are all the internal threads in the Java process, as shown in the following figure:

We will see that the proportion of each thread is about the same, and occasionally a certain thread is relatively high. when some threads account for a higher proportion, this small example will end up with a higher proportion of garbage collection threads, because garbage collection can not come over. So you need to keep recycling back and forth, a little bit every time, in fact, this kind of example is very likely to be your business logic thread. The proportion of business logic threads in that area is very high, so it's time to use another command-- jstack.

3 、 jstack

When we use top-Hp to know which thread it is, we can next use the jstack command, for example, if we want to check the thread number 31083, 31061 is our process PID, and we want to locate a thread whose cpu proportion is much higher than other cpu, then we need to locate what kind of problem is in this thread, we need to write down this thread number (31083).

Because the thread number used in jstack is hexadecimal, we need to convert 31083 decimal to hexadecimal.

Features:

Each thread has its own thread number, which contains the status of the thread. You can observe whether the thread is blocked. If a long period of wait and block indicates that there is something wrong with the thread.

4. Convert hexadecimal

Because the thread ID in the Java thread file is hexadecimal, you need to convert the thread ID from decimal to hexadecimal command: echo "obase=16;31083" | bc

5. Jstack usage parsing [root@root ~] # jstackUsage: jstack [- l] (to connect to running process) jstack-F [- m] [- l] (to connect to a hung process) jstack [- m] [- l] (to connect to a core file) jstack [- m] [- l] [server_id@] (to connect to a remote debug server) Options:-F to force a thread dump. Use when jstack does not respond (process is hung)-m to print both java and native frames (mixed mode)-l long listing. Prints additional information about locks-h or-help to print this help message6, jstack view output

We can also use jps or java ps-ef | java to view the Java process. Here we use jps to view the process

[root@root ~] # jps

[root@root ~] # jstack 31061

"pool-1-thread-3" # 10 prio=5 os_prio=0 tid=0x00007f3568105800 nid=0x7961 waiting on condition [0x00007f35455cf000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park (Native Method)-parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park (LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await (AbstractQueuedSynchronizer.java:2039 ) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask (ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) at java.lang.Thread.run (Thread .java: 748) "pool-1-thread-2" # 9 prio=5 os_prio=0 tid=0x00007f3568103800 nid=0x7960 waiting on condition [0x00007f35456d0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park (Native Method)-parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park (LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask (ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) at java.lang. Thread.run (Thread.java:748) "pool-1-thread-1" # 8 prio=5 os_prio=0 tid=0x00007f3568102000 nid=0x795f waiting on condition [0x00007f35457d1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park (Native Method)-parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park (LockSupport.java:175) at java.util.concurrent.locks .AbstractQueuedSynchronizer $ConditionObject.await (AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take (ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask (ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) At java.lang.Thread.run (Thread.java:748) "Service Thread" # 7 daemon prio=9 os_prio=0 tid=0x00007f35680b4000 nid=0x795d runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C1 CompilerThread1" # 6 daemon prio=9 os_prio=0 tid=0x00007f35680b1000 nid=0x795c waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" # 5 daemon prio=9 os_prio=0 tid=0x00007f35680af000 nid=0x795b waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" # 4 daemon prio=9 os_prio=0 tid=0x00007f35680ad800 nid=0x795a runnable [0x0000000000000000 ] java.lang.Thread.State: RUNNABLE "Finalizer" # 3 daemon prio=8 os_prio=0 tid=0x00007f356807c800 nid=0x7959 in Object.wait () [0x00007f3558301000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait (Native Method)-waiting on (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:144)-locked (a java.lang.ref.ReferenceQueue$Lock ) at java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:165) at java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:216) "Reference Handler" # 2 daemon prio=10 os_prio=0 tid=0x00007f3568077800 nid=0x7958 in Object.wait () [0x00007f3558402000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait (Native Method)-waiting on (a java.lang.ref.Reference$Lock) At java.lang.Object.wait (Object.java:502) at java.lang.ref.Reference.tryHandlePending (Reference.java:191)-locked (a java.lang.ref.Reference$Lock) at java.lang.ref.Reference$ReferenceHandler.run (Reference.java:153) "main" # 1 prio=5 os_prio=0 tid=0x00007f3568009800 nid=0x7956 waiting on condition [0x00007f356ed59000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep ( Native Method) at FullGCTest.main (FullGCTest.java:35) "VM Thread" os_prio=0 tid=0x00007f356806e000 nid=0x7957 runnable "VM Periodic Task Thread" os_prio=0 tid=0x00007f35680b7000 nid=0x795e waiting on condition JNI global references: 205

Analyze thread state through thread dump

In most cases, the current operation of each thread will be analyzed based on thead dump, such as whether there is a deadlock, whether there is a thread holding the lock for a long time, and so on.

In dump, threads generally have the following states: 1, RUNNABLE, thread is in execution 2, BLOCKED, thread is blocked 3, WAITING, thread is waiting

Locked indicates that the thread locks the 0x000000076bf62208 object; waiting to lock indicates that the thread is waiting for the lock on the 0x000000076bf62208 object; waiting for monitor entry [0x000000001e21f000] indicates that the thread enters the critical section of the monitor through the synchronized keyword and is in the "Entry Set" queue, waiting for monitor; waiting on (a java.lang.Object) to release the lock.

If there are 100 threads in a process, many threads are in waiting on a lock, and then the thread should not be blocked, the end is not over, we must find which thread holds the lock, we can search the jstack dump information, find the information to see which thread has only this lock, generally the thread state is RUNNABLE, indicating that the thread is running but has been holding the lock. Then it will lead to the deadlock of the whole thread.

7. Jstack analysis deadlock public class TestDeadLock {private static Object obj1 = new Object (); private static Object obj2 = new Object (); public static void main (String [] args) {new Thread (new Thread1 ()). Start (); new Thread (new Thread2 ()). Start () } private static class Thread1 implements Runnable {@ Override public void run () {synchronized (obj1) {System.out.println ("Thread1 has got obj1's lock!") The significance of try {/ / pausing for 2 seconds is to let the Thread2 thread get obj2's lock Thread.sleep (2000);} catch (InterruptedException e) {e.printStackTrace ();} synchronized (obj2) {System.out.println ("Thread1 has got obj2's lock!") ;} private static class Thread2 implements Runnable {@ Override public void run () {synchronized (obj2) {System.out.println ("Thread2 has got the lock of obj2!") The significance of try {/ / pausing for 2 seconds is to let the Thread1 thread get the lock Thread.sleep (2000) of obj1;} catch (Exception e) {e.printStackTrace () } synchronized (obj1) {System.out.println ("Thread2 got obj1's lock!") ;}

View the analysis log through the command

[root@root fuccGC] # jps485 Bootstrap9877 Jps10629 QuorumPeerMain9846 TestDeadLock [root@root fuccGC] # jstack 9846

Use of memory monitoring tools

We can use the command that comes with jvm to monitor GC information: jinfo pid: this command is to list some details of the process [root@root] # jinfo 9846 this is just helpful, but not very helpful, you just have to remember this command, do not do in-depth understanding of jstat-gc pid 1000: this is to print out the GC log every second. Dynamic observation of GC situation / reading GC log found frequent GC, etc., but this information does not seem to be very intuitive, and there are not many things that can be analyzed, so it is not generally used.

What we use most is to check through tools, such as jconsole/jvisualvm.

1 、 jconsole

These two tools are included with JDK, and they are also graphical interface tools. As long as you install JDK, you can track a process on a remote server locally. As a Linux server, few people install a graphical interface, as shown below:

Add parameters when our program starts:

Java-Djava.rmi.server.hostname=101.XX.XXX.XX-Dcom.sun.management.jmxremote-Dcom.sun.management.jmxremote.port=8080-Dcom.sun.management.jmxremote.ssl=false-Dcom.sun.management.jmxremote.authenticate=false-Dcom.sun.management.jmxremote.rmi.port=8080 FullGCTest

The basic situation is as follows, we can monitor the memory information on our remote server.

2 、 jvisualvm

Double-click jvisualvm

Select remote-Click the File button

Choose to add a JMX connection and enter our address and account password to log in.

So we can view the memory information of our remote service.

Here we know how to locate the problem. Which class takes up how much memory will be displayed. Click on the sampler, memory, and then the JAVA process memory of the remote machine will be shown in the graphical interface, how many classes are there and how many bytes are occupied, so that we can specifically locate which class has a problem.

How to answer the location memory overflow (OOM) in the interview

If the interviewer asks us how to locate OOM, but we can't answer it with a graphical interface, because as a service, it will show the original efficiency of the service when we open a JMX service, then what does the system that we have launched need not be graphically used? There is also a graphical interface called Jprofiler, which is best used, but this one is free, so it is generally not needed.

If we don't use graphical interfaces, what can we use? where can we use cmdline and arthas graphical interfaces? Used in testing, monitoring during testing ~

If the project is already online, what can we use without a graphical interface? We can use Jmap.

Jmap

[root@root ~] # jmap-histo 19086 | head-20

It will print out our memory information, although it is not as convenient as the graphical interface, but the information inside is enough for us to observe and locate the problem.

Online systems with large memory will have a great impact on the process during the execution of jmap, even stutter (not suitable for e-commerce)

A heap dump file (java-Xms20M-Xmx20M-XX:+UseParallelGC-XX:+HeapDumpOnOutOfMemoryError FullGCTest) is automatically generated when the parameter HeapDump,OOM is set.

Many servers are backed up (highly available). Stopping this server will not affect other servers.

Thank you for your reading, the above is the content of "how to troubleshoot the problem of excessive JVM memory". After the study of this article, I believe you have a deeper understanding of how to troubleshoot the problem of excessive JVM memory, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.