Shulou(Shulou.com)06/02 Report--

Today, I would like to talk to you about the process of solving weird bug. Many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something from this article.

This article records a weird discovery process of bug (the process is inexplicably terminated), and details the reason behind the problem, as well as the solution.

Background

This project is an interface automation testing platform, which needs to be connected to the automation project (each automation project corresponds to a tested application project), and each automation project is an independent process. Automation project uses springboot, using maven plug-in to run, mvn spring-boot:run. The platform is connected with many automation projects, and many automation processes are running on a linux machine.

Phenomenon

Occasionally, when the automation project is killed, it needs to be re-operated on the platform to start the automation project before you can continue to run the task. Until the scale of the system expands, the number of automation projects becomes more, the automation use cases become more, the running task time becomes longer, and the probability that the automation project is inexplicably terminated increases.

Analysis.

Check log and find that the log information of the terminated automation project is as follows:

[INFO]-[INFO] BUILD FAILURE [INFO]- [INFO] Total time: 09:32 min [INFO] Finished at: 2019-08-15T17:39:55+08:00 [INFO]-- -[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.2.0.M5:run (default-cli) on project coupon_batch-Test: Application finished with exit code: 137-> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors Re-run Maven with the-e switch. [ERROR] Re-run Maven using the-X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Analyze the causes of log, google: ERROR: Maven JVM terminated unexpectedly with exit code 137

For reference the status code 137128 + 9 typically means (can differ between flavours of unix). That the process was terminated by receipt of a signal. In this case signal 9 which is SIGKILL and unblockable kill. If this is the case the underlying machine/OS needs more virtual memory. This can be added by either adding physical memory or swap space as appropriate. You should try to increase the virtual memory of your machine. Note:This also explains why a Jenkins restart (temporarily) fixes the issue. Jenkins Maven Build 137 Error

At this point, we found the cause of the problem: insufficient linux swap partitions, which caused the process to be killed by linux OOM Killer.

Background knowledge needed to solve the problem 1. What is swap?

A swap space is an area on disk that can be a partition, a file, or a combination of them. To put it simply, when the system's physical memory is tight, Linux will save the infrequently accessed data in memory to swap, so that the system will have more physical memory to serve each process, and when the system needs to access the content stored on swap, it will load the data on swap into memory, which is what we often call swap out and swap in.

two。 Why do I need swap?

For some large applications (such as LibreOffice, video editor, etc.), a large amount of memory is used during startup, but this memory is often only used at startup, and this memory is rarely used later in the run process. With swap, the system can save this part of the memory data that is not used in this way to swap, thus releasing more physical memory for the system to use.

The hibernation function of many distributions (such as ubuntu) depends on the swap partition. When the system sleeps, the data in memory will be saved to the swap partition, and the data will be loaded into memory the next time the system starts, which can speed up the startup of the system. So if you want to use the hibernation function, you must configure the swap partition, and the size must be greater than or equal to the physical memory.

In some cases, what if you want to run a program that consumes memory but has limited physical memory? At this point, you can achieve your goal by configuring enough swap space, which is slower, but at least you can run.

Although in most cases, physical memory is sufficient, there are always some unexpected situations, such as a process needs more memory than expected, or a process has a memory leak. When there is not enough memory, the OOM killer of the kernel will be triggered. According to the configuration of OOM killer, some processes will be dropped by kill or restarted directly by the system (the default is to give priority to the one that consumes the most memory). However, with swap, we can use swap as memory, which is a bit slow, but at least gives us a chance to go to the debug, kill process or save the current work progress.

If you have looked at Linux memory management, you will know that the system will use as much free memory as possible for cache to speed up the system's ILinux O speed, so if you can move less commonly used memory data to swap, there will be more physical memory for cache, thus improving the overall performance of the system.

3. Shortcomings of swap

The advantages of swap are described above, but what about the disadvantages of swap? Swap is stored on the disk, and the speed of the disk is several orders of magnitude slower than the memory. If you keep reading and writing swap, it will definitely affect the performance of the system. Especially when the system memory is very tight, the frequency of reading and writing swap space will be very high, causing the system to run very slowly. Like dead, adding physical memory is the only solution.

Because the system will automatically move uncommonly used memory data to swap, for desktop programs, it may cause a small card to be opened after minimizing a program, because the data on swap needs to be reloaded into memory.

Linux swap space (swap space)

Solution: add swap file # first create a new 512m file to be used as a swap file, the file path can be arbitrary # fallocate this command depends on the file system, some old file systems do not support this command, such as ext2 # in this case, you can use dd to achieve the same effect: the difference between # sudo dd if=/dev/zero of=/mnt/512MiB.swap bs=1024 count=524288#fallocate and dd is that # fallocate declares so much first, and then the file system allocates the real physical disk space only when it is specifically used, that is, to allocate a little bit, while dd actually writes 512m of data to the physical disk space from the very beginning. # so fallocate is more convenient as a test, because you don't have to write any data at first, but you have to dev@dev:~$ sudo fallocate-l 512m / mnt/512MiB.swap# to modify the permission of the file. To prevent other users from mismanipulating this file, dev@dev:~$ sudo chmod 600 / mnt/512MiB.swap# is formatted as swap file dev@dev:~$ sudo mkswap / mnt/512MiB.swap# to add new files to the system dev@dev:~$ sudo swapon / mnt/512MiB.swap#. When you can see that the new swap file has been added to the system, the type is file#. Here you can see that due to the highest priority The first swap partition / dev/dm-1 has been used with 24Kdevdevdevavo $swapon-sFilename Type Size Used Priority/dev/dm-1 partition 524284 24-1/dev/sdb1 partition 2096124 0-2/mnt/512MiB.swap file 524284 0-partition as you can see from the output of the free command, adding swap partitions and files after the first two rounds # now the switching space of the system has become 3G (3144692K) dev@dev:~$ free total used free shared buff/cache availableMem: 500192 39112 9564 1996 451516 430820Swap: 3144692 24 314466 430820Swap also to ensure that our new swap files will be loaded automatically after the system restart Need to modify / etc/fstab file dev@dev:~$ sudo sh-c 'echo "/ mnt/512MiB.swap none swap sw 0" > > / etc/fstab' Advanced: OOM Killer configuration

Understand and configure OOM Killer under Linux

There are always reasons for technical problems, depending on whether you can find them or not. If you delve into the problem of technology, there are many places to learn, and an bug is a learning opportunity. After reading the above, do you have any further understanding of the process of solving weird bug? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.