Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to troubleshoot the online Full GC derived from the double 12 pressure test in java". In the daily operation, I believe that many people have doubts about how to troubleshoot the online Full GC derived from the double 12 pressure test in java. The editor consulted all kinds of data and sorted out a simple and useful method of operation. I hope it will be helpful for you to answer the doubt of "how to troubleshoot the online Full GC derived from the double 12 pressure test in java". Next, please follow the editor to study!

Online problem

Double 12 before the pressure test from a very small amount, directly triggered the Full GC, scared urine, because the double 12 to promote preheating is about to begin, this may not be a good 3.25.

Quickly pull the group, pull the relevant students together to check the problem.

Check the GC log for the first time:

You can see that the reason is that the threshold of Metadata GC has been exceeded, triggering the recovery of Full GC,Metaspace from 243m to 231m, which is hardly recycled, so with a little more volume, it is easy to trigger the recycling of Metaspace again.

Knowledge reserve

GC troubleshooting requires a lot of reserve knowledge, the most important is related to JVM, the previous article has already talked about some, here mainly talk about what is Matespace? Later, I will talk about how to do the troubleshooting of GC problems.

Here are two points of knowledge:

What is Matespace (metaspace)? What role do you play in JVM, that is, what does it store? What does Full GC have to do with the Matespace size setting?

Matespace is called metaspace, and since JDK 8, the concept of PermGen has been abandoned and replaced by a storage space called Metaspace.

Metaspace is used to store: the runtime data structure of Class files in JVM, as well as other Klass-related contents, such as Method,ConstantPool, etc.

Metaspace uses local memory, not heap memory, which means that by default the size of Metaspace is only related to the size of local memory. But it is easy to misunderstand that Matespace can be used at will, no matter how much is used, as long as it does not exceed the local memory, GC will not be triggered, which is wrong.

The setting method of Matespace is:-XX: MetaspaceSize=**M. The function of this JVM parameter is to trigger Full GC when Matespace reaches MetaspaceSize. If Matespace is not set, the default value is very small, about 20m (different systems will be a little different). If the program has more Load Class, it is easy to trigger Full GC. What is important to understand here is that both the Class information and the ClassLoader that loads the Class are stored in Metaspace, and we know that a class is uniquely determined by the class loader of this class plus a fully qualified name (consisting of package name & class name).

So you can check the size of your own application JVM Metaspace setting, if not, you can check the default value through-XX:+PrintFlagsInitial.

(the previous article has sent a detailed explanation of the GC log and the configuration instructions of the JVM parameters. If you have any questions, you can take a look at the history article. )

Problem troubleshooting

At first, I saw Full GC frequently, and I checked the log because there was not enough space in Metaspace. The first reaction was to adjust the size of Metaspace, increasing the MetaspaceSize from 256m to 512m. However, it is found that the Full GC caused by Metaspace has not been eliminated.

Immediately dump the logs of the two machines, analyze the GC log file for the first time, and find no abnormality. Here is a note that the timing of the dump file is very important. Sometimes the GC log of dump has no problem because it happens to be dump after the Full GC is completed, and the memory recovery is clean. Some problems of slow memory increase must be dump before Full GC.

During this period, we also found that the cache-related objects occupy more memory, but after analysis, the life cycle of the cache object itself is relatively long, so it is resident on the heap, there is no problem, continue to look.

Inspection found that the Metaspace memory footprint increased with the growth of the double 12 new interface pressure test traffic, so it can be determined that the new interface code was introduced.

An analysis of the GC dump log shows that nearly 3000 are loaded on the same ClassCloader, as shown in the following figure. As we said earlier, the ClassCloader information is in the Metaspace area.

The case was solved, and the improper use of fastjson introduced the pit of ASM Serializer.

Fault location & repair

The reason why FastJson is fast is that it uses asm bytecode enhancement instead of reflection. So it must be the frequent loading of the same class when the ASM of fastjson is applied in the code to deal with the data. the basic problem is that the code is masturbated and the problem code is quickly located.

Because of confidentiality, it is not convenient to put the original code, Angela made a similar logic code:

For (Item item-> arrays) {

SerializeConfig serializeConfig = new SerializeConfig ()

SerializeConfig.put (Custom.class, new CustomSerializer ())

JsonString = JSON.toJSONString (item, serializeConfig)

}

This code customizes a serialization implementation class CustomSerializer, which performs an AOP-like aspect processing when serializing, uniformly customizing all fields of Custom type (using CustomSerializer).

In principle, the implementation of the need for serialization of Class using asm to dynamically generate a proxy class, in this case is the Item class, using SerializeConfig configuration to dynamically generate bytecode proxy class: com.alibaba.fastjson.serializer.ASMSerializer_1_Item, but each new SerializeConfig object, will be regarded as each proxy goal is different, resulting in each time a new class loader ASMClassLoader will be used, so Metaspace will soon be full, triggering frequent Full GC.

If you want to study further, you can take a look at the FastJson source code:

Com.alibaba.fastjson.serializer.SerializeConfig#createASMSerializer

Problem fix:

Register ObjectSerializer to make ObjectSerializer globally unique, which is equivalent to a singleton.

SerializeConfig.getGlobalInstance () .put (Character.class, new CharacterSerializer ()

When SerializeConfig is changed to singleton, each serialization does not have to be loaded repeatedly.

Similar troubleshooting & tuning if the dump log finds that many classloader names have the same prefix, check to see if there is any use of this dynamic proxy technology, and may be constantly generating proxy objects. It is found that memory grows slowly, GC cannot be recycled, dump GC logs are checked to see if any classes are loaded repeatedly; Metaspace tuning, for example, our current production environment Metaspace basically sets 256m or 512m, which can be determined according to the type of application and machine memory configuration, factors: 1. Whether more classes will be loaded, 2. The memory of the machine allows, and the Metaspace can be adjusted appropriately. At this point, the study on "how to troubleshoot the online Full GC derived from the double 12 pressure test in java" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.