Shulou(Shulou.com)06/02 Report--

Author: kelvinjin2009 Source: programmer

Original link: http://www.techug.com/post/java-and-docker-memory-limits.html

Java and Docker are not natural friends. Docker can set memory and CPU limits, which Java cannot automatically detect. Using Java's Xmx logo (tedious / repetitive) or the new experimental JVM logo, we can solve this problem.

Mismatch in Virtualization

The combination of Java and Docker is not a perfect match. At first, there is a considerable distance from the perfect match. For beginners, the whole idea of JVM is that virtual machines can make programs independent of the underlying hardware.

So what are the benefits of packaging our Java application into a JVM and then stuffing the whole into an Docker container? In most cases, you are just copying JVMs and Linux containers, which does no good except to waste more memory. I feel stupid like this.

However, Docker can package your programs, settings, specific JDK,Linux settings and application servers, as well as other tools as one thing. From the perspective of DevOps/Cloud, such a complete container has a higher level of encapsulation.

Problem 1: memory

Today, the vast majority of production applications still use Java 8 (or older), which can cause problems. Java 8 (pre-update 131s) and Docker don't work well together. The problem is that the amount of available memory and CPU of JVM on your machine is not the amount of available memory and CPU that Docker allows you to use.

For example, if you limit your Docker container to 100MB memory, however, older versions of Java do not recognize this limitation. Java does not see this limit. JVM will require more memory and far exceed that limit. If too much memory is used, Docker will take action and kill the processes in the container! The JAVA process was killed, which is obviously not what we wanted.

To solve this problem, you need to specify a maximum memory limit for Java. In the old version of Java (before 8u131), you needed to limit the heap size by setting-Xmx in the container. This doesn't feel right, and you don't want to define these limits twice, nor do you want to define them in your container.

Fortunately, we now have a better way to solve this problem. After Java 9 (8u131 +), JVM has added the following flags:

-XX:+UnlockExperimentalVMOptions-XX:+UseCGroupMemoryLimitForHeap

These flags force JVM to check the cgroup configuration of the Linux, and Docker achieves the maximum memory setting through cgroup. Now, if your application reaches the limit set by Docker (such as 500MB), JVM can see this limit. JVM will try the GC operation. If the memory limit is still exceeded, JVM does what it has to do and throws an OutOfMemoryException. That is, JVM can see these settings for Docker.

After Java 10 (see the test below), these experience flags are on by default, or you can use-XX:+UseContainerSupport to enable them (you can disable these behaviors by setting-XX:-UseContainerSupport).

Question 2: CPU

The second problem is similar, but it has to do with CPU. In short, JVM will look at the hardware and detect the number of CPU. It will optimize your runtime to use these CPUs. But in the same case, there is another mismatch, and Docker may not allow you to use all of these CPUs. Unfortunately, this was not fixed in Java 8 or Java 9, but it was resolved in Java 10.

Starting with Java 10, available CPUs calculations will solve this problem in a different way (by default) (again through UseContainerSupport).

Memory processing Test of Java and Docker

As an interesting exercise, let's verify and test how Docker uses several different JVM versions / flags and even different JVM to handle memory shortages.

First, we create a test application that simply "eats" memory and does not release it.

Javaimport java.util.ArrayList;import java.util.List;public class MemEat {public static void main (String [] args) {List l = new ArrayList (); while (true) {byte b [] = new byte [1048576]; l.add (b); Runtime rt = Runtime.getRuntime (); System.out.println ("free memory:" + rt.freeMemory ()) }}}

We can start the Docker container and run the application to see what happens.

Test 1: Java 8u111

First, we'll start with containers with older versions of Java 8 (update 111s).

Shelldocker run-m 100m-it java:openjdk-8u111 / bin/bash

We compile and run the MemEat.java file:

Shelljavac MemEat.javajava MemEat...free memory: 67194416free memory: 66145824free memory: 65097232Killed

As expected, Docker has killed our Java process. It's not what we want. ). You can also see the output, which Java thinks still has a lot of memory to allocate.

We can solve this problem by using the-Xmx flag to provide maximum memory for Java:

Shelljavac MemEat.javajava-Xmx100m MemEat...free memory: 1155664free memory: 1679936free memory: 2204208free memory: 1315752Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at MemEat.main (MemEat.java:8)

After providing our own memory limit, the process stops normally, and JVM understands the limit that it is running. The problem, however, is that you now set these memory limits twice, Docker once and JVM once.

Test 2: Java 8u144

As mentioned earlier, with the addition of new flags to fix the problem, JVM can now follow the settings provided by Docker. We can test it with a newer version of JVM.

Shelldocker run-m 100m-it adoptopenjdk/openjdk8 / bin/bash

(at the time of this writing, the version of this OpenJDK Java image is Java 8u144)

Next, we compile and run the MemEat.java file again without any flags:

Shelljavac MemEat.javajava MemEat...free memory: 67194416free memory: 66145824free memory: 65097232Killed

The same problem still exists. But we can now provide the experimental signs mentioned above to give it a try:

Shell

Javac MemEat.java

Java-XX:+UnlockExperimentalVMOptions-XX:+UseCGroupMemoryLimitForHeap MemEat

...

Free memory: 1679936

Free memory: 2204208

Free memory: 1155616

Free memory: 1155600

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

At MemEat.main (MemEat.java:8)

This time we didn't tell JVM what the limit was, we just told JVM to check the correct limit setting! I feel much better now.

Test 3: Java 10u23

Some people mentioned in comments and Reddit that Java 10 solves all problems by making the lab flag the new default value. This behavior can be turned off by disabling this flag:-XX:-UseContainerSupport.

When I tested it, it didn't work at first. At the time of this writing, the AdoptAJDK OpenJDK10 image is packaged with jdk-10+23. The JVM obviously still doesn't understand the UseContainerSupport flag, and the process is still killed by Docker.

Shell

Docker run-m 100m-it adoptopenjdk/openjdk10 / bin/bash

Tested the code (and even manually provided the required flags):

Shelljavac MemEat.javajava MemEat...free memory: 96262112free memory: 94164960free memory: 92067808free memory: 89970656Killedjava-XX:+UseContainerSupport MemEatUnrecognized VM option 'UseContainerSupport'Error: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit.

Test 4: Java 10u46 (Nightly)

I decided to try the latest nightly build for AdoptAJDK OpenJDK 10. It contains a version of Java 10: 46, not Java 10: 23.

Shelldocker run-m 100m-it adoptopenjdk/openjdk10:nightly / bin/bash

However, there is a problem in this ngithly build that the exported PATH points to the old Java 10: 23 directory instead of 10: 46, which we need to fix.

Shellexport PATH=$PATH:/opt/java/openjdk/jdk-10+46/bin/javac MemEat.javajava MemEat...free memory: 3566824free memory: 2796008free memory: 1480320Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at MemEat.main (MemEat.java:8)

Success! Java 10 still correctly detects the Dockers memory limit without providing any flags.

Test 5: OpenJ9

I've also been experimenting with OpenJ9 recently, and this free alternative JVM has been open source from IBM J9 and is now maintained by Eclipse.

Please read more about OpenJ9 in my next post (http://royvanrijn.com/blog/2018/05/openj9-jvm-shootout/).

It runs fast, has excellent memory management and excellent performance, and can often save up to 30-50% memory for our micro services. This almost defines a Spring Boot application as' micro', which runs for only 100-200mb instead of 300mb+. I intend to write an article on this as soon as possible.

But to my surprise, OpenJ9 doesn't have an option similar to the flag (backported) for cgroup memory limits in Java 8 / 9 / 10 +. If we apply the previous test case to the latest AdoptAJDK OpenJDK 9 + OpenJ9 build:

Shelldocker run-m 100m-it adoptopenjdk/openjdk9-openj9 / bin/bash

Let's add the OpenJDK flag (the flag that OpenJ9 will ignore):

Shelljava-XX:+UnlockExperimentalVMOptions-XX:+UseCGroupMemoryLimitForHeap MemEat...free memory: 83988984free memory: 82940400free memory: 81891816Killed

Oops,JVM was killed by Docker again.

I really hope that a similar option will be added to OpenJ9 soon, because I want to run this option in a production environment without having to specify a maximum memory twice. Eclipse/IBM is working hard to fix this problem by mentioning issues and even submitting a PR for issues.

Update: (Hack is not recommended)

A slightly ugly / hacky way to solve this problem is to use the following combination flag:

Shelljava-Xmx`cat / sys/fs/cgroup/memory/memory.limit_in_ bytes` MemEat...free memory: 3171536free memory: 2127048free memory: 2397632free memory: 1344952JVMDUMP039I Processing dump event "systhrow" Detail "java/lang/OutOfMemoryError" at 14:04:26 on 2018-05-15-please wait.JVMDUMP032I JVM requested System dump using'/ / core.20180515.140426.125.0001.dmp' in response to an eventJVMDUMP010I System dump written to / / core.20180515.140426.125.0001.dmpJVMDUMP032I JVM requested Heap dump using'/ / heapdump.20180515.140426.125.0002.phd' in response to an eventJVMDUMP010I Heap dump written to / / heapdump.20180515.140426.125.0002.phdJVMDUMP032I JVM requested Java dump using'/ / javacore.20180515. 140426.125.0003.txt' in response to an eventJVMDUMP010I Java dump written to / / javacore.20180515.140426.125.0003.txtJVMDUMP032I JVM requested Snap dump using'/ / Snap.20180515.140426.125.0004.trc' in response to an eventJVMDUMP010I Snap dump written to / / Snap.20180515.140426.125.0004.trcJVMDUMP013I Processed dump event "systhrow" Detail "java/lang/OutOfMemoryError" .exception in thread "main" java.lang.OutOfMemoryError: Java heap space at MemEat.main (MemEat.java:8)

In this case, the heap size is limited by the memory allocated to the Docker instance, which applies to older JVM and OpenJ9. This is, of course, wrong, because the container itself and other parts of the JVM outside the heap also use memory. But it seems to work, and it is clear that Docker is loose in this case. Maybe some bash gods will make a better version, subtracting some of the bytes from other processes.

In any case, don't do this, it may not work properly.

Test 6: OpenJ9 (Nightly)

Some people suggest using the latest nightly version of OpenJ9.

Shelldocker run-m 100m-it adoptopenjdk/openjdk9-openj9:nightly / bin/bash

The latest nightly version of OpenJ9 has two things:

Another problematic PATH parameter, which needs to be solved first.

JVM supports the new flag UseContainerSupport (just like Java 10)

Shellexport PATH=$PATH:/opt/java/openjdk/jdk-9.0.4+12/bin/javac MemEat.javajava-XX:+UseContainerSupport MemEat...free memory: 5864464free memory: 4815880free memory: 3443712free memory: 2391032JVMDUMP039I Processing dump event "systhrow" Detail "java/lang/OutOfMemoryError" at 21:32:07 on 2018-05-15-please wait.JVMDUMP032I JVM requested System dump using'/ / core.20180515.213207.62.0001.dmp' in response to an eventJVMDUMP010I System dump written to / / core.20180515.213207.62.0001.dmpJVMDUMP032I JVM requested Heap dump using'/ / heapdump.20180515.213207.62.0002.phd' in response to an eventJVMDUMP010I Heap dump written to / / heapdump.20180515.213207.62.0002.phdJVMDUMP032I JVM requested Java dump using'/ / javacore.20180515. 213207.62.0003.txt' in response to an eventJVMDUMP010I Java dump written to / / javacore.20180515.213207.62.0003.txtJVMDUMP032I JVM requested Snap dump using'/ / Snap.20180515.213207.62.0004.trc' in response to an eventJVMDUMP010I Snap dump written to / / Snap.20180515.213207.62.0004.trcJVMDUMP013I Processed dump event "systhrow" Detail "java/lang/OutOfMemoryError" .exception in thread "main" java.lang.OutOfMemoryError: Java heap space

TADAAA, it's under repair!

Oddly enough, this flag is not enabled by default in OpenJ9, just as it is in Java 10. Say it again: make sure you test that you want to run Java in a Docker container.

Conclusion

In short: pay attention to the mismatch of resource constraints. Test your memory settings and JVM flags, and don't assume anything.

If you are running Java in a Docker container, make sure that you set Docker memory limits and do so in JVM, or that your JVM understands these limits.

If you cannot upgrade your version of Java, use-Xmx to set your own limits.

For Java 8 and Java 9, update to the latest version and use:

-XX:+UnlockExperimentalVMOptions-XX:+UseCGroupMemoryLimitForHeap

For Java 10, make sure it supports' UseContainerSupport' (update to the latest version).

For OpenJ9 (which I strongly recommend, which can effectively reduce memory footprint in a production environment), use-Xmx to set the limit now, but there will soon be a version that supports the UseContainerSupport flag.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.