Shulou(Shulou.com)06/03 Report--

Troubleshooting application memory leaks

1. The origin of the article

In the process of daily operation and maintenance, we will encounter the high server resources or the oom caused by the soaring memory of CPU, which leads to the unavailability of services (most programs are java applications). This article is written to provide a reference basis for work troubleshooting and ways to quickly locate problems.

two。 Basic knowledge reserve

(1). Jvm configuration common parameters:

Heap parameter parameter

Parameters.

Description-Xms sets the initialization size of heap memory when JVM starts-Xmx sets the maximum heap memory-Xmn sets the space size of the younger generation, and the rest is the space size of the older generation

-XX:PermGen sets the memory initialization size of the permanent generation (JDK1.8 begins to discard the permanent generation)

-XX:MaxPermGen sets the maximum value of the permanent generation-XX:SurvivorRatio sets the space ratio of the Eden area and the Survivor area: Eden/S0 = Eden/S1 default 8-XX:NewRatio sets the ratio between the older generation and the younger generation, and the default value is 2 recycler parameter-XX:+UseSerialGC

Serial, young (young area) and Old (old area) all use serial, use replication algorithm to recover, logic is simple and efficient, no field switching overhead-XX:+UseParallelGC parallel, young (young area) uses Parallel scavenge recovery algorithm, multiple threads will be reclaimed in parallel. Specify the number of threads through the-XX:ParallelGCThreads=n parameter, which defaults to the number of cpu cores. Old: single thread-XX:+UseParallelOldGC parallelism is the same as UseParallelGC. Both young (young zone) and Old (elderly zone) garbage collection use multi-thread collection-XX:+UseConMarkSweepGC

For concurrent, short-paused concurrent collection, the young area can use a normal or parallel garbage collection algorithm, which is controlled by the parameter-XX:+UseParNewGC; the Old area only uses Concurrent Mark Sweep-XX:+UseG1GC

Parallel, concurrent, and incremental short-paused garbage collectors. There is no distinction between young (young area) and Old (old area) space. It divides the heap space into multiple areas of equal size. When carrying out garbage collection, he will give priority to collecting areas with fewer surviving objects. So it's called "Garbage First."

As shown in the table above, there are mainly serial, parallel and concurrent. For applications with large memory, the performance of serial is too low, so parallel and concurrency are mainly used. The policies for parallel and concurrent GC are specified by UseParallelGC and UseConcMarkSweepGC, and there are some detailed configuration parameters used to configure how the policy is executed. For example: XX:ParallelGCThreads, XX:CMSInitiatingOccupancyFraction, etc. Generally speaking, object collection in Young area can only choose parallelism (time-consuming), and concurrency in Old area (consumption CPU).

You can refer to common configurations in the project.

Production environment service configuration parameters

System version

Cpumemjdk version jvm configuration remarks

CentOS 7.5.1804 (Core)

8C

8G

Java-1.8.0_121

Java-Xmx5g-Xms5g-Xmn3g-XX:MetaspaceSize=32m-XX:MaxMetaspaceSize=128m-XX:MaxDirectMemorySize=512m-XX:+UseCompressedClassPointers-XX:CompressedClassSpaceSize=64m-XX:+UseConcMarkSweepGC-XX:ParallelCMSThreads=2-XX:+CMSClassUnloadingEnabled-XX:+UseCMSCompactAtFullCollection-XX:CMSInitiatingOccupancyFraction=80-verbose:gc-XX:+PrintGCDateStamps-XX:+PrintGCTimeStamps-XX:+PrintGCDetails-XX:+PrintTenuringDistribution-XX:+PrintCommandLineFlags-XX:+ExplicitGCInvokesConcurrent-Xloggc:/data/log/app-web/gc.log-Dapp.id=app-web-Dapollo.bootstrap.enabled=true-Duser.timezone=GMT+8-Dapollo.bootstrap.namespaces=application-Deureka.instance.metadata-map .zone = zone-1-Dservice.name=app-web-Denv=prod-Dprod_meta= http://configserver-prod.com-Druntime.env=prod-jar. / lib/app-web-1.8.10.0.jar

Micro service

Springcloud framework

Common combination

GC tuning principle

Before tuning, we need to keep the following principles in mind:

Most Java applications do not require GC optimization on the server

Most Java applications that cause GC problems are not because of our wrong parameter settings, but because of code problems.

Before launching the application, consider setting the JVM parameters of the machine to the optimal (most suitable)

Reduce the number of objects created

Reduce the use of global variables and large objects

GC optimization is the last resort.

In practical use, analyzing GC optimization code is much more than optimizing GC parameters.

GC tune the target

Minimize the number of objects transferred to the old era

Reduce the execution time of GC.

Strategy 1: keep the new object in the new generation. Because the cost of Full GC is much higher than that of Minor GC, it is wise to allocate objects to the new generation as far as possible. In the actual project, according to the GC log, analyze whether the space allocation of the new generation is reasonable, and adjust the size of the new generation through the "- Xmn" command to minimize the situation that the new object directly enters the old era.

Strategy 2: large objects are entering the old age, although in most cases, it makes sense to allocate objects to the new generation. However, this practice for large objects is open to question. If large objects are allocated for the first time in the new generation, there may be a lack of space in the old era when many small objects under age are allocated, destroy the object structure of the new generation, and there may be frequent full gc. Therefore, for large objects, you can set them directly into the old age (of course, short-lived large objects are always a nightmare for garbage collection). -XX:PretenureSizeThreshold can set the size of objects that go directly into the old age.

Strategy 3: reasonably set the age of the object entering the old age,-XX:MaxTenuringThreshold sets the age of the object to enter the old age, reduce the memory consumption of the old age, and reduce the frequency of full gc.

Strategy 4: set a stable heap size with two parameters:-Xms initializes the heap size and-Xmx maximum heap size.

Strategy 5: note: GC optimization is generally not required if the following metrics are met:

MinorGC execution time is less than 50ms

Minor GC is executed infrequently, about once in 10 seconds.

Full GC execution time is less than 1s.

The frequency of Full GC execution is not frequent, not less than once in 10 minutes.

3. Investigation skill war

1. Techniques for troubleshooting memory leaks

System command

1. Log in to the probe server, first of all, the third company of top free df

Jstat

Jstat is a very powerful JVM monitoring tool, and its general usage is: jstat [- options] pid interval

The view items it supports are:

-class to view class loading information

-compile compilation statistics

-gc garbage collection information

-gcXXX details of GC in each region, such as-gcold

3. Jstat-gc pid [interval] View the GC status of the java process

Follow indicator: FULL GC.

4. Jstack pid > jstack.log query thread stack and save the site

Analysis stack

Stack analysis is simple to see if there are too many threads and what most stacks are doing.

> grep 'java.lang.Thread.State' jstack.log | wc-l

> 464

Filtering process

Grep-A 1 'java.lang.Thread.State' jstack.log | grep-v' java.lang.Thread.State' | sort | uniq-c | sort-n

10 at java.lang.Class.forName0 (Native Method)

10 at java.lang.Object.wait (Native Method)

16 at java.lang.ClassLoader.loadClass (ClassLoader.java:404)

44 at sun.nio.ch.EPollArrayWrapper.epollWait (Native Method)

344 at sun.misc.Unsafe.park (Native Method)

There seems to be no exception in the thread state, and then analyze the heap file.

Memory heap dump

5. Use jmap-dump:format=b,file=heap.log pid to save the heap site, and then restart the application service

The heap files are all binary data, so it is very troublesome to view them on the command line. The tools provided by Java are all visual, but you can't view them on the Linux server, so you have to download the files locally first.

Because we set the heap memory to 4G, the heap file from dump is also very large, so it is really troublesome to download it, but we can compress it first. =

Gzip is a very powerful compression command, especially we can set-1 ~-9 to specify its compression level, the larger the data, the greater the compression ratio, the longer the time, it is recommended to use-6x7,-9 is too slow, and the benefit is not great, with this compression time, the extra files can be downloaded.

Using MAT to analyze jvm heap

MAT is a powerful tool for analyzing Java heap memory, use it to open our heap file (change the file suffix to .hprof), it will prompt us to analyze the type, decisively choose memory leak suspect.

The windows system installation package is as follows:

Https://www.eclipse.org/downloads/download.php?file=/mat/1.9.1/rcp/MemoryAnalyzer-1.9.1.20190826-win32.win32.x86_64.zip

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.