Shulou(Shulou.com)06/02 Report--

Linux performance memory part one, memory management

The ​ linux kernel provides each process with an independent virtual address space that is contiguous. In this way, the process can easily access memory, that is, virtual memory.

​ virtual address space is divided into kernel space and user space, and the range of address space is also different for processors with different word length (the maximum length that cpu instructions can process data). Such as 32-bit and 64-bit

When the ​ process is in user mode, it can only access the memory in user space; only when it enters the kernel state can it access kernel space memory. Although the address space of each process contains kernel space, these kernel spaces are actually associated with the same physical memory. In this way, after the process switches to the kernel state, it can easily access the kernel space memory.

​ every process has such a large address space, so the virtual memory of all processes is naturally much larger than the actual physical memory. Not all virtual memory allocates physical memory, only those that are actually used allocate physical memory, and the allocated physical memory is managed by memory mapping.

​ memory mapping is actually mapping virtual memory addresses to physical memory addresses. To complete memory mapping, the kernel maintains a page table for each process, recording the mapping between virtual addresses and physical addresses

The ​ page table is actually stored in cpu's memory management unit MMU, so that under normal circumstances, the processor can find the memory to access directly through the hardware. When the virtual address accessed by the process cannot be found in the page table, the system will generate a page fault exception, enter the kernel space to allocate physical memory, update the process page table, and finally return to user space to resume the process.

​ TLB is the cache of page tables in MMU. Because the virtual address space of the process is independent, and the access speed of TLB is much faster than that of MMU, by reducing the context switching of the process and the refresh times of TLB, the utilization of TLB cache can be improved, and then the memory access performance of cpu can be improved.

Instead of managing memory in bytes, ​ MMU specifies the smallest unit of memory mapping, that is, the page size 4KB. Each memory mapping requires an integer multiple of memory space associated with 4KB or 4KB.

​ to solve the problem of too many items in the page table, linux provides two mechanisms, namely, multi-level pages and large pages (HugePage).

​ multilevel pages: is to divide the memory into blocks to manage, changing the original mapping relationship into block indexes and offsets within blocks. Since only a small portion of the virtual memory space is usually used, the multi-level page saves these blocks in use, which greatly reduces the number of items in the page table.

​ linux uses a four-level page table to manage memory pages. The first four table items are used to select the page, and the last index represents the intra-page offset.

​ large pages: larger blocks of memory than normal pages. Common ones are 2MB and 1GB. Large pages are usually used on processes that use a lot of memory, such as Oracle, etc.

2. How linux processes use memory

Virtual memory space distribution

1. Read-only segments: including code and constants

2. Data segments: including global variables

3. Heap: including dynamic memory allocation, growing upward from low address

4. File mapping segment: including dynamic libraries, shared memory, etc., growing downwards from high address

5. Stack: including local variables and the context of function calls. The size of the stack is fixed, usually 8MB.

Memory allocation and recycling

​ malloc () is a memory allocation function provided by the C standard library, corresponding to system calls, there are two ways to implement: brk () and mmap ().

​ brk (): small chunks of memory (less than 128k). The C standard library uses brk () to allocate memory, that is, by moving the top of the heap to allocate memory. This memory is not immediately returned to the system after it is freed, but is cached so that it can be reused.

​ MMap (): large chunks of memory (larger than 128k) are allocated directly using memory mapping mmap (), that is, a piece of free memory is allocated in the file mapping segment and returned to the system directly after release.

​, however, these two allocations have their advantages and disadvantages. Brk can reduce the occurrence of page fault exceptions and improve memory access efficiency. This memory is not returned to the system, when the memory is busy, frequent memory allocation and release will cause memory fragmentation. Mmap returns the system directly, so a page fault exception occurs every time mmap. When memory is busy, frequent memory allocation will lead to a large number of page fault exceptions, which will increase the administrative burden of the kernel.

​ linux reclaims memory in three ways:

Reclaim cache: recycle the least recently used memory pages using the LRU algorithm

Recycle infrequently accessed memory: write infrequently used memory directly to disk through swap partitions

Kill processes: when memory is tight, the system will directly kill processes that take up a lot of memory through oom.

In the second way, the swap partition swap is used when reclaiming infrequently accessed memory. Swap is actually using a piece of disk space as memory. It can store data that is not temporarily used by the process to disk (this process is swapping out). When the process accesses the memory, it reads the data from disk into memory (this process is called swapping in). Swap is usually used only when there is insufficient memory.

The third way, oom, is actually a protection mechanism for the kernel. It monitors the memory usage of processes and uses oom_score to score the memory usage of each process:

The more memory a process consumes, the larger the oom_score

The more cpu a process uses to run, the smaller the oom_score will be.

​ can adjust the oom_score of a process by manually setting its oom_adj. The range of oom_adj [- 17 oom 15]. The higher the value, the easier it is for the process to be killed by oom; the smaller the value is, the less likely it is for the process to be killed by oom, where-17 means that oom is prohibited.

[root@test proc] # lsof-i:22COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAMEsshd 3043 root 3U IPv4 21799 0t0 TCP *: ssh (LISTEN) sshd 3499 root 3u IPv4 23448 0t0 TCP test:ssh- > 123.139.156.118IPv4 53251 (ESTABLISHED) sshd 3552 root 3u IPv4 23652 0t0 TCP test:ssh- > 123.139.156.118IPv4 53254 (ESTABLISHED) [root@test proc] # cat / proc/3043/oom_adj-"at this time the oom_adj of the ssh process is-17 Ssh is not easy to kill.

Check the overall memory condition

Free

[root@test] # free-hm total used free shared buff/cache availableMem: 991M 93M 399M 484K 498m 737MSwap: 0B 0B 0B [root@test] # grep Cached / proc/meminfo Cached: 408672 kBSwapCached: 0 kB [root@test] # grep Buffer / proc/meminfo Buffers: 41252 kB

Total: total memory size

Used: the amount of memory already used, including shared memory

Free: unused memory size

Shared: shared memory size

Buff/cache: size of cache and buffer

Available: the amount of memory available for the new process.

Note: available contains recyclable caches, so it is larger than the unused memory of free. Not all caches can be reclaimed.

Top

[root@test proc] # toptop-11:42:36 up 2:10, 2 users, load average: 0.00,0.01, 0.05Tasks: 77 total, 1 running, 76 sleeping, 0 stopped, 0 zombie%Cpu (s): 0.3 us, 0.3 sy, 0.0 ni, 98.0 id, 1.3 wa, 0.0 hi, 0.0 si, 0.0 stKiB Mem: 1015024 total, 352724 free, 94188 used 568112 buff/cacheKiB Swap: 0 total, 0 free, 0 used. 747996 avail Mem PID USER PR NI VIRT RES SHR S% CPU% MEM TIME+ COMMAND 3476 root 20 0 611432 13556 2364 S 0.3 1.3 0 barad_agent 3775 root 20 571352 7168 2528 S 0.3 0.7 root 10.34 YDService 1 root 20 0 125476 3916 2592 S 0.0 0.4 0:01.45 systemd 2 root 20 00 00 S 0.0 0.00: 00.00 kthreadd 3 root 20 00 00 S 0.0 0.00: 00.10 ksoftirqd/0 5 root 0-20 000 S 0.0 0.00: 00.00 kworker/0:0H 7 root Rt 00 00 S 0.0 0.00: 00.00 migration/0 8 root 20 00 00 S 0.0 0.00: 00.00 rcu_bh 9 root 20 00 00 S 0.0 0.0 0:00.61 rcu_sched

VIRT: the size of the virtual memory of a process that is counted as long as it has been applied, even if physical memory has not been actually allocated.

RES: resident memory size, that is, the physical memory actually used by the process, but excluding swap and shared memory

SHR: the size of shared memory, such as shared memory used with other processes, loaded dynamic link libraries, and code snippets of the program.

% MEM: the process uses physical memory as a percentage of system memory.

Buffers and Cached

Buffers is the temporary storage of the original disk block, that is, the data used to cache the disk, by not being very large (dozens of MB). In this way, the kernel can centralize scattered writes, uniformly optimize disk writes, and merge multiple small writes into single large writes.

Cached is a page cache that reads files from disk, that is, to cache data read from files. In this way, the next time it is accessed, it is read directly from memory.

SReclaimable is part of Slab. Slab consists of two parts, recyclable: SReclaimable. Non-recyclable: SUnreclaim.

Note: buffers and cache have both read and write. Not a single read or write.

Process cache hit ratio

Bcc package is required to view cache hit ratio

Centos7 system installation

[root@centos ~] # yum update

[root@centos] # rpm-- import

Https://www.elrepo.org/RPM-GPG-KEY-elrepo.org & & rpm-Uvh

Http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

[root@centos ~] # uname-r # #

3.10.0-862.el7.x86_64

[root@centos ~] # yum remove kernel-headers kernel-tools kernel-tools-libs

[root@centos ~] # yum-- disablerepo= "--enablerepo=" elrepo-kernel "install

Kernel-ml kernel-ml-devel kernel-ml-headers kernel-ml-tools

Kernel-ml-tools-libs kernel-ml-tools-libs-devel

[root@centos] # sed-I'/ GRUB_DEFAULT/s/=./=0/' / etc/default/grub

[root@centos] # grub2-mkconfig-o / boot/grub2/grub.cfg

[root@centos ~] # reboot

[root@centos ~] # uname-r # # upgraded successfully

4.20.0-1.el7.elrepo.x86_64

[root@centos ~] # yum install-y bcc-tools

[root@centos ~] # echo 'export PATH=$PATH:/usr/share/bcc/tools' > / etc/profile.d/bcc-tools.sh

[root@centos] #. / etc/profile.d/bcc-tools.sh

[root@centos ~] # cachestat 1 1 # # Test whether the installation is successful

Ubuntu system installation steps

Sudo apt-key adv-- keyserver keyserver.ubuntu.com-- recv-keys 4052245BD4284CDDecho "deb https://repo.iovisor.org/apt/xenial xenial main" | sudo tee / etc/apt/sources.list.d/iovisor.listsudo apt-get updatesudo apt-get install-y bcc-tools libbcc-examples linux-headers-$ (uname-r) # bcc is installed in the / usr/share/bcc/tools directory. The path path export PATH=$PATH:/usr/share/bcc/tools of the system needs to be configured

Note: bcc package, must be kernel 4.1 or above.

Cache hit of cachestat/cachetop process

Root@VM-16-7-ubuntu:~# cachestat 1 3 HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB 3414 0 5 100.00% 32 563 59 0 4 100.00% 32 563 62 0 4 100.00% 32 563root@VM-16-7-ubuntu:~# cachetop 12:50:16 Buffers MB: 80 / Cached MB: 572 / Sort: HITS / Order: ascendingPID UID CMD HITS MISSES DIRTIES READ_HIT% WRITE_HIT% 3267 root YDService 200 100.0% 1843 root barad_agent 3 0 1 66.7% 0.0% 20042 root barad_agent 8 03 62.5% 0.02% 323 root jbd2/vda1-8 8 8 6 12.5% 12.5% 20041 root barad_agent 9 2 4 45.5% 9.1% 20034 root cachetop 300 0 0 100.0% 0.0% 20044 root sh 15800 100.0% 20045 root sh 15800 100.0% 0.0% 20046 root sh 15800 100.0% 0% 20043 root sh 39900 100.0% 20044 root cat 49900 100.0% 0.00% 20043 root barad_agent 49600 100.0% 0.00% 20045 root Grep 638 00 100.0% 0.00% 20046 root awk 91500 100.0% # indicator TOTAL: total number of IO MISSES: the number of cache misses; HITS: cache hits; DIRTIES: the number of dirty pages added to the cache; the size of CACHED_MB:buffer, in units of MB; the size of BUFFERS_MB:cache, in units of MB

Pcstat file cache view

# pcstat installation: if [$(uname-m) = "x86y64"] Then curl-L-o pcstat https://github.com/tobert/pcstat/raw/2014-05-02-01/pcstat.x86_64else curl-L-o pcstat https://github.com/tobert/pcstat/raw/2014-05-02-01/pcstat.x86_32fichmod 755 pcstat./pcstat # can use # root@VM-16-7-ubuntu:~# lspcstatroot@VM-16-7-ubuntu:~#. / pcstat pcstat |- -- +-| | Name | Size | Pages | Cached | Percent | |-+-- -+-| | pcstat | 3049296 | 745 | 100.000 | |-+-| # # indicator 3. Memory leak

​ when a process requests virtual memory through malloc (), the system does not immediately allocate physical memory to it. Instead, it traps into the kernel to allocate memory through a page fault exception when it is first accessed. To reconcile the performance differences between fast cpu and slow disks, linux also uses cache and buffer to cache data read and written to files and disks into memory, respectively. Therefore, for programs, dynamic memory allocation and recycling is the location of the accident.

The allocated memory was not reclaimed correctly, resulting in a leak.

The address outside the allocated memory boundary is accessed, causing the program to exit abnormally, and so on.

Allocation and recovery of memory

Memory space of the process

1. Read-only paragraphs: including code and constants. No new memory will be allocated, so there will be no memory leak

2. Data segment: including global variables. Variables are sized when they are defined, so there is no memory leak

3. Heap: including dynamic allocation of memory, growing upward from a low address. Application allocation management, did not release heap memory correctly, memory leak

4. File mapping segment: including dynamic libraries, shared memory, etc., growing downward from the high address. Shared memory is also program management, memory leak

5. Stack: including local variables and the context of function calls. The size of the stack is fixed, usually 8MB. System management, no leakage

IV. Memory leak cases

Environment

2cpu 8GB memory

Pre-install sysstat docker bcc

# install docker, Sysstatsudo apt-get install-y sysstat docker.io# install bccsudo apt-key adv-- keyserver keyserver.ubuntu.com-- recv-keys 4052245BD4284CDDecho "deb https://repo.iovisor.org/apt/bionic bionic main" | sudo tee / etc/apt/sources.list.d/iovisor.listsudo apt-get updatesudo apt-get install-y bcc-tools libbcc-examples linux-headers-$ (uname-r) # pull image docker run-- name=app-itd feisk / app:mem-leak # k followed by Y # check root@test:~# docker logs app2th = > 13th = > 24th = > 35th = > 56th = > 87th = > 138th = > 219th = > 3410th = > 55

Investigation

Vmstat

Root@test:~# vmstat 3procs-memory- swap---io-----system---cpu- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 143036 108584 558084 20 63 146 313 10 99 143036 142904 108584 558068 00 00 173 400 00 100 00 00 0 142440 108588 558072 00 0 271 159 333 11 96 2 00 00 142472 108588 558072 00 0 17 128 313 10 99 00 100 142472 108588 558072 00 0 5 115 283 00 100 00.... .... 00 142412 108596 558076 00 029 297 708 1 98 00 00 141480 108628 558152 00 0 12 170 404 00 99 00 0 141512 108628 558152 00 04 17290 00 100 00 0141512 108628 558152 00 016 176 399 1 98 00 # observation for a period of time, it was found that free decreased continuously, while buffer and cache remained basically unchanged. Indicates that the system memory is in use all the time, but the memory leak cannot be explained.

Memleak

In the bcc package

Root@test:~# / usr/share/bcc/tools/memleak-a-p 17050Attaching to pid 17050, Ctrl+C to quit.cannot attach uprobe Device or resource busy [19:26:43] Top 10 stacks with outstanding allocations: addr = 7f389c2fc700 size = 8192 addr = 7f389c2f86e0 size = 8192 addr = 7f389c300720 size = 8192 addr = 7f389c2fa6f0 size = 8192 addr = 7f389c2fe710 size = 8192 40960 bytes in 5 allocations from stack fibonacci+0x1f [app] child+0x4f [app] start_thread+0xdb [libpthread-2.27.so] [19:26:48] Top 10 stacks with outstanding allocations:#-an indicates that each The size of the memory allocation request and address #-p the PID number # / usr/share/bcc/tools/#-an of the specified case application indicates the size of each memory allocation request and the address #-p specifies the PID number # app process of the case application has been allocating memory And the memory allocated by the fibonacci () function is not freed. = # check code root@test:~# docker exec app cat / app.c#include # include long long * fibonacci (long long * n0, long long * N1) {long long * v = (long long *) calloc (1024, sizeof (long long)); * v = * n0 + * N1; return v;} void * child (void * arg) {long long n0 = 0; long long N1 = 1; long long * v = NULL; for (int n = 2; n > 0) ) {v = fibonacci (& n0, & N1); n0 = N1; N1 = * v; printf ("% dth = >% lld\ n", n, * v); sleep (1);} int main (void) {pthread_t tid; pthread_create (& tid, NULL, child, NULL); pthread_join (tid, NULL); printf ("main thread exit\ n"); return 0 } root@test:~# # found that child () called the fibonacci function, but did not free the memory returned by fibonacci. Add free (v) to the child function; free memory

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.