Shulou(Shulou.com)06/03 Report--

This article mainly introduces "how to understand the case troubleshooting process". In the daily operation, I believe many people have doubts about how to understand the case troubleshooting process. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to understand the case troubleshooting process". Next, please follow the editor to study!

Concrete process

Failed to apply for 4G memory

As shown in the figure above, the record shows that the request for 4G memory failed (4294967296 B / 1024 / 1024 = 4096 M).

Is min_free_kbytes & nr_hugepage misconfigured? The first reaction is to recall that the free caused by vm.min_free_kbytes & nr_hugepage is greater than that of available.

Memfree counts the free memory of all memory, while memavailable counts the memory that can be used by the program, and the customer sets vm.min_free_kbytes (2.5G), which contains free statistics, but without memavailable statistics, nr_hugepage will also have this problem.

The statistical methods of the two are different. For more information, please see Documentation/filesystems/proc.txt.

MemFree: The sum of LowFree+HighFreeMemAvailable: An estimate of how much memory is available for starting new applications, without swapping. Calculated from MemFree, SReclaimable, the size of the file LRU lists, and the low watermarks in each zone. The estimate takes into account that the system needs some page cache to function well, and that not all reclaimable slab will be reclaimable, due to items being in use. The impact of those factors will vary from system to system. Ask the customer for information analysis questions such as free-m & & sysctl-p & & / proc/meminfo. HugePages_Total is 0, which means that nr_hugepage is not set. MemAvailable: 7418172 kB, indicating that so much memory is available. # sysctl-p

Net.ipv4.ip_forward = 0

Net.ipv4.conf.default.accept_source_route = 0

Kernel.sysrq = 1

Kernel.core_uses_pid = 1

Net.ipv4.tcp_syncookies = 1

...

Net.ipv4.tcp_tw_recycle=1

Net.ipv4.tcp_max_syn_backlog=4096

Net.core.netdev_max_backlog=10000

Vm.overcommit_memory=2

...

# cat / proc/meminfo

MemTotal: 8009416 kB

MemFree: 7347684 kB

MemAvailable: 7418172 kB

Buffers: 18924 kB

Cached: 262836 kB

SwapCached: 0 kB

Active: 315188 kB

Inactive: 222364 kB

Active (anon): 256120 kB

Inactive (anon): 552kB

Active (file): 59068 kB

Inactive (file): 221812 kB

....

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

DirectMap4k: 114560 kB

DirectMap2M: 4079616 kB

DirectMap1G: 6291456 kB

Try to recreate the attempt to test by yourself using the java command to apply for more than my tester's physical memory and get an error.

In fact, the above meminfo has already explained the problem, but due to lack of experience, I don't understand what's going on for the time being.

The following test proves that there will be no problem with the normal application of memory, and the excess memory will be OOM.

[root@test] # java-Xms4096M-version

Openjdk version "1.8.0mm 242"

OpenJDK Runtime Environment (build 1.8.0_242-b08)

OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)

[root@test] # java-Xms5000M-version

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory (0x0000000687800000, 3495428096, 0) failed; error='Cannot allocate memory' (errno=12)

.

The system information is as follows:

-S Y S T E M-

OS:CentOS Linux release 7.4.1708 (Core)

Uname:Linux 3.10.0-693.2.2.el7.x86_64 # 1 SMP Tue Sep 12 22:26:13 UTC 2017 x86 million 64

Libc:glibc 2.17 NPTL 2.17

Rlimit: STACK 8192k, CORE 0k, NPROC 15088, NOFILE 65535, AS infinity

Load average:0.05 0.05 0.05

/ proc/meminfo:

MemTotal: 3881692 kB

MemFree: 2567724 kB

MemAvailable: 2968640 kB

Buffers: 69016 kB

Cached: 536116 kB

SwapCached: 0 kB

Active: 355280 kB

Inactive: 326020 kB

...

VmallocTotal: 34359738367 kB

VmallocUsed: 14280 kB

VmallocChunk: 34359715580 kB

HardwareCorrupted: 0 kB

AnonHugePages: 30720 kB

HugePages_Total: 256

HugePages_Free: 256

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

DirectMap4k: 57216 kB

DirectMap2M: 3088384 kB

DirectMap1G: 3145728 kB

....

Memory: 4k page, physical 3881692k (2567600k free), swap 0k (0k free)

Vm_info: OpenJDK 64-Bit Server VM (25.242-b08) for linux-amd64 JRE (1.8.0_242-b08), built on Jan 28 2020 14:28:22 by mockbuild "with gcc 4.8.5 20150623 (Red Hat 4.8.5-39)

Time: Thu Feb 20 15:13:30 2020

Timezone: CST

Elapsed time: 0 seconds (0d 0h 0ms)

Repeat failed, continue to analyze the Java test to prove that the normal application of memory will not be a problem, excess memory will be OOM, so why the excess, the line of sight returned to sysctl-p found something.

Vm.overcommit_memory=2

About the overcommit_memory setting item:

Overcommit_memory=0

By default, the kernel does a check when the application process tries to request memory. The kernel will check if there is enough memory available for the application process

If there is enough memory available, the memory request is allowed; otherwise, the memory request fails and the error is returned to the application process.

For example, for a 1G machine, process A has already used 500m. When another process tries malloc 500m of memory, the kernel will check and find that it exceeds the remaining available memory and will prompt a failure.

Overcommit_memory=1

For memory request, the kernel will not do any check until the physical memory is used up, triggering the OOM kill-user process.

Also in the above example, the 1G machine, A process 500m and process B try malloc 500m, which will succeed, but once kernel finds that the memory usage is close to 1 G (the kernel has a policy), it will trigger OOM and kill some user-mode processes (strategic killing).

Overcommit_memory=2

When the requested memory > = SWAP memory size + physical memory * N, the memory request is rejected. Explain that the NRV N is a percentage, determined by overcommit_ratio/100, such as overcommit_ratio=50 (my test machine defaults to 50%), then N is 50%. Vm.overcommit_ratio takes effect only when vm.overcommit_memory = 2. The memory that can be applied for is SWAP memory + physical memory * overcommit_ratio/100.

Take a look at the overcommit information in the log above:

CommitLimit: 4004708 kB (less than 4096m requested by the customer) Committed_AS: 2061568 kB

Specifically:

CommitLimit: the maximum memory that can be allocated (it takes effect when vm.overcommit_memory=2 is tested). The specific values are: SWAP memory size (ecs is not open) + physical memory * overcommit_ratio / 100 overcommit_ratio committedmemory as: the amount of memory currently allocated

5. The comparison indicates that the vm.overcommit_memory set by the customer is in effect. It is recommended to change it back to 0 and try again.

Failed to allocate memory with vm.overcommit_memory = 2 test; [root@test ~] # grep-I commit / proc/meminfo

CommitLimit: 1940844 kB

Committed_AS: 480352 kB

# java-Xms2048M-version failed

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory (0x0000000080000000, 1431830528, 0) failed; error='Cannot allocate memory' (errno=12)

#

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (mmap) failed to map 1431830528 bytes for committing reserved memory.

# An error report file with more information is saved as:

# / root/hs_err_pid1267.log

You can restore it with the following configuration: vm.overcommit_memory = 0, vm.overcommit_ratio = 50#vm.overcommit_memory = 0

# vm.overcommit_ratio = 50

[root@test] # java-Xms2048M-version

Openjdk version "1.8.0mm 242"

OpenJDK Runtime Environment (build 1.8.0_242-b08)

OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode) at this point, the study on "how to understand the case investigation process" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.