Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about how to understand the OOMKiller of the Java process. Many people may not know much about it. In order to make you understand better, the editor has summarized the following for you. I hope you can get something according to this article.

01 OOM killer

After logging on to the machine, check the application and middleware log, and you really don't see the problem. I suspected JVM OOM but did not configure the output. I was about to add the stack output parameters of OOM, but I found that the application startup command already contained a similar parameter:-XX:+HeapDumpOnOutOfMemoryError-XX:ErrorFile=./hs_err_pid.log-XX:HeapDumpPath=./java_pid.hprof. It seems that nothing can be found from the application level, so let's take a look at the system log. Use dmesg-T | grep java to view the system log. Sure enough, from the information returned on the penultimate line of dmesg, it is indeed caused by oom, but the oom here is not the oom of JVM, but the oom of the Linux system. > Killed process xxx (java), total-vm:7539096kB, anon-rss:3919048kB file-rss:17312kB, shmem-rss:0kB.

On this machine with 4GB physical memory, `total- vm` is the amount of virtual memory that has been allocated to the java process (7539096kb=7.02GB) `anon- rss` is the java process physical memory usage (3919048kB=3.65GB) `file- rss` is the amount of java process mapped to the device or file system (17312kB=16.51MB)

There have been many articles about Linux oom killer on the Internet, and here is a simple summary: when the Linux system allocates physical memory, if there is insufficient memory (when is it insufficient? Oom killer will choose the process with the highest oom score score to kill to free memory.

02 JVM OOM vs. Linux OOM

It seems clear that it was Linux's oom killer that killed the Java process. But I still have a question, also OOM, why not trigger the OOM of JVM? With this doubt in mind, I went on to learn about Linux's memory allocation mechanism.

02.1 overcommit and oom killer

To briefly explain my understanding, the real distribution mechanism is much more complicated. Please correct anything that is wrong. Linux has a virtual memory mechanism. When a process requests memory from the system, the system can reply to the process in two ways: a. Check whether there is enough memory (the actual value is generally equal to Swap+RAM*overcmmit_ratio) to meet the demand, if so, to meet the allocation, if not, to reply to the integration with allocation failure; b. No inspection, direct distribution.

The allocation here refers to the growth of the virtual memory address of the process. The b here is called "Overcommit", that is, an allocation that exceeds the range of memory acceptable to the system. The b here is actually based on an optimistic estimate, because the requested memory is not necessarily used. But once a process needs to use the actual memory it can provide, it results in OOM, and oom killer frees up memory by sorting oom score at the expense of one or more of the highest-scoring processes.

Once again, the key point of understanding Overcommit is that memory allocation is not equal to memory allocation, and memory allocation is only when memory is used.

02.2 JVM OOM

What about JVM's OOM?

> One common indication of a memory leak is the java.lang.OutOfMemoryError exception. Usually, this error is thrown when there is insufficient space to allocate an object in the Java heap. In this case, The garbage collector cannot make space available to accommodate a new object, and the heap cannot be expanded further. Also, this error may be thrown when there is insufficient native memory to support the loading of a Java class. In a rare instance, a java.lang.OutOfMemoryError may be thrown when an excessive amount of time is being spent doing garbage collection and little memory is being freed. Excerpt from Oracle memory leak description related articles

To put it simply, the reasons for JVM OOM are: insufficient memory in the Java heap, the garbage collector cannot make room for new objects, and the heap cannot be expanded (Java heap space); local memory is not enough to support loading Java classes (Metaspace or Perm); and uncommonly, too long garbage collection does not free up much memory will also lead to OOM (GC Overhead limit exceeded). Looking back, in fact, the Linux system "cheated" the Java process. When the policy to allow overcommit is turned on, if the Java process or any other process requests possibly too much virtual memory (more than the system can provide), OOM will not occur in the Java process as long as the system memory is sufficient. When the Linux system finds that there is really not enough memory allocated, it will quietly kill the Java process at the top of the oom score (kill-9).

> Java farmer: "Master, give me some land. I want to grow melons." Master Linux: "I'm glad to give you 4096 mu of land. Do whatever you want!" Java old farmer farming. 1. 2. 4. 1024. Java farmer: "I'm so happy without bug harassment. I want to be like this forever." Master Linux, the Java pawn: "does anyone want land? my lord, I have plenty of land!"

03 solution

After analyzing the reasons, there are actually many solutions.

03.1 request a machine with more memory

If you have the resources, try the easiest way first.

03.2 optimize the code

Today's machines are generally containers or virtual machines as long as there is only one application running, if OOM occurs, the most fundamental solution is to return to the code.

03.3 prohibit Overcommit

Let the system report errors in the application, conservative and safe. Expose the problem as soon as possible and fix it as soon as possible. At the same time, you can change the value of overcommit_ratio to change CommitLimit. See Appendix for configuration.

03.4 prohibit OOM Killer from killing critical application processes

For some machines where multi-application processes are mixed, key processes can be protected from being dropped by kill. Sudo echo-1000 > / proc/$pid/oom_score_adj

03.5 prohibit OOM Killer

Modify the panic_on_oom parameter to turn off OOM Killer. It's okay to play, but the online system is not recommended, otherwise the system will die and show you.

Appendix: 1 Strategy of Overcommit for Linux

Can be configured or viewed at / proc/sys/vm/overcommit_memory.

> 0: default value. Heuristic strategy, the more serious Overcommit will not be satisfied, while the slight Overcommit will be allowed. In addition, the value of root Overcommit is slightly higher than the average user, generally 3%. 1: allow Overcommit, this strategy is suitable for applications that cannot afford memory allocation failures, such as some scientific computing applications. (this strategy is enabled by the system involved in this article) 2: disable Overcommit. In this case, the memory allocated by the system will not exceed the following CommitLimit. The calculation method is Swap + RAM * / proc/sys/vm/overcmmit_ratio. The default is 50%. If so many resources have been used up, any subsequent attempt to apply for memory will return an error. This usually means that no new programs can be run at this time.

2 View allocated memory grep-I commit / proc/meminfoCommitLimit: 6201492 kB # Virtual memory limit Committed_AS: 5770836 kB # allocated memory. If it is greater than CommitLimit, it means that Overcommit is enabled. 3 View the oom score of oom-related # processes. 0 is not compatible with killcat / proc/ {pid} / oom_score # process oom adjustment. Oom_score_adj is generally used to replace cat / proc/ {pid} / oom_adj # users in the calculation. The minimum value of-1000 will prohibit oom killer from killing this process. # values range from-1000 to 1000, indicating the discount to penalty for the final score. Cat / proc/ {pid} / oom_score_adj after reading the above, do you have any further understanding of how to understand the OOMKiller of the Java process? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.