Shulou(Shulou.com)06/01 Report--

What this article shares with you is about the relationship between memory leak and memory overflow in big data and what the solution is. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article. Say no more, let's take a look at it with the editor.

I. memory leak

Memory leak memory leak means that the program cannot release the applied memory space after applying for memory. The harm of a memory leak can be ignored, but the consequence of memory leak accumulation is very serious. No matter how much memory it is, it will be occupied sooner or later.

Second, memory overflow

Memory overflow out of memory means that when the program applies for memory, there is not enough memory space for its use. When out of memory; appears, for example, the number that can be saved only after applying for an integer but saving long for it, that is, memory overflow.

Third, the relationship between the two

Memory leak (memory leak) will eventually lead to out of memory (memory overflow)!

Memory overflow: just like a plate using all kinds of methods can only hold 4 fruits, you install 5, and you fall to the ground and can't eat it. This is spillover. For example, the stack, when the stack is full, it is bound to produce space overflow, which is called upper overflow, and when the stack is empty, it also produces space overflow, which is called underflow. That is, the allocated memory is not enough to drop the sequence of data items, which is called memory overflow. To put it bluntly, I can't bear so much, so I'm wrong.

Memory leak: means that you apply to the system for allocation of memory for use (new), but do not return it after use (delete). As a result, you can no longer access the piece of memory you applied for (maybe you lost its address), and the system cannot allocate it to the needed program again. It is equivalent to renting a cabinet with a key. after you lock the cabinet after you have saved something, you lose the key or do not return the key, then the result is that the cabinet will not be available to anyone and cannot be recycled by the garbage collector. because I can't find any information about him.

Fourth, according to the way it occurs, memory leaks can be divided into four categories:

1. Frequent memory leaks. Code that has a memory leak is executed multiple times, resulting in a memory leak each time it is executed.

2. Occasional memory leak. Code that has a memory leak occurs only in certain environments or operations. Recurrent and sporadic are relative. For a particular environment, sporadic may become frequent. Therefore, the test environment and test methods are very important to detect memory leaks.

3. One-time memory leak. The code that has a memory leak will only be executed once, or because of an algorithmic flaw, there will always be a leak of only one piece of memory. For example, memory is allocated in the constructor of a class, but not freed in the destructor, so a memory leak occurs only once.

4. Implicit memory leak. The program keeps allocating memory while it is running, but it doesn't release memory until the end. Strictly speaking, there was no memory leak here, because in the end, the program freed all the requested memory. But for a server program, it takes days, weeks, or even months to run, and not releasing memory in time can eventually deplete all the memory in the system. Therefore, we call this type of memory leak an implicit memory leak.

From the point of view of the user using the program, the memory leak itself will not cause any harm, as the general user, can not feel the existence of the memory leak at all. What is really harmful is the accumulation of memory leaks, which will eventually consume all the memory of the system. From this point of view, one-time memory leaks are harmless because they do not accumulate, while implicit memory leaks are very harmful because they are more difficult to detect than recurrent and sporadic memory leaks.

Fifth, the causes of memory overflow and its solutions:

Cause of memory overflow:

1. The amount of data loaded in memory is too large, such as fetching too much data from the database at once

2. There is a reference to the object in the collection class, which is not cleared after use, so that the JVM cannot be recycled.

3. There is an endless loop in the code or an object entity whose loop produces too much repetition.

4. BUG in the third-party software used

5. The memory value of startup parameters is set too small.

Solution to memory overflow:

The first step is to modify the JVM startup parameters to directly increase memory. (don't forget to add the parameter-Xms,-Xmx.)

The second step is to check the error log to see if there are other exceptions or errors before the "OutOfMemory" error.

The third step is to walk through and analyze the code to find out where the memory overflow may occur.

Focus on the following points:

1. Check whether there is a query to get all the data in the database query. Generally speaking, if 100, 000 records are taken to memory at a time, it may cause a memory overflow. This problem is more hidden, before going online, there is less data in the database, so it is not easy to cause problems. After going online, there is more data in the database, and a query may cause memory overflow. Therefore, the database query should be paged as far as possible.

two。 Check the code for endless loops or recursive calls.

3. Check to see if there are large loops that repeatedly generate new object entities.

4. Check whether there is a query to get all the data in the database query. Generally speaking, if 100, 000 records are taken to memory at a time, it may cause a memory overflow. This problem is more hidden, before going online, there is less data in the database, so it is not easy to cause problems. After going online, there is more data in the database, and a query may cause memory overflow. Therefore, the database query should be paged as far as possible.

5. Check whether there are any problems that have not been cleared after the collection objects such as List and MAP are finished. Collection objects such as List, MAP, etc., always have references to objects, so that these objects cannot be recycled by GC.

The above is the relationship between memory leak and memory overflow in big data and what is the solution. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.