Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "what is the method of Linux virtual memory management". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

5. Is the memory of free really released (back to OS)?

A serious problem with all the previous examples is that the allocated memory is not released, which leads to a memory leak. In principle, all memory allocated by malloc/new needs to be freed by free/delete. But is the free memory really freed?

To clarify this problem, you can illustrate it through the following example.

Initial state: as shown in figure (1), the system has allocated four blocks of ABCD memory, in which ABD is allocated in the heap and C uses mmap. For simplicity, the address space of file mapping areas such as shared libraries is ignored in the figure.

E=malloc (100k): allocates 100k memory, less than 128k, allocates from the heap, there is not enough space left in the heap, and extends the brk pointer.

Free (A): frees up A's memory, which in glibc is simply marked as available, creating a memory hole (fragment) that is not really freed. If the space within 40k needs to be allocated at this time, the space can be reused and the remaining space forms new small fragments.

Free (C): C space is greater than 128K, allocated with mmap, if C is released, munmap system call is called to release the space, and the space is actually released to OS, as shown in figure (4).

Free (D): similar to releasing A, releasing D will also cause a hole and get free space, but it will not be returned to OS. At this time, the total free space is 100K, but due to the discontiguous virtual address, it can not be merged, so the free space can not meet the allocation request of more than 60k.

Free (E): release E, because it is continuous with D, the two will be merged to get 160k continuous free space. At the same time, E is the space closest to the heap top. In the free implementation of glibc, as long as the total space (including merged space) near the heap top exceeds 128k, sbrk (- SIZE) will be called to trace the stack top pointer and return the original heap top space to OS, as shown in figure (6). The free space in the heap will not be returned to OS.

Thus it can be seen that:

Malloc uses the memory allocated by mmap (greater than 128k), and free calls the munmap system call to return it to OS immediately for real release.

For the memory in the heap, only when the space on the top of the heap is freed and the total continuous free space on the top of the heap is greater than 128k will sbrk (- SIZE) be used to reclaim the memory and really return the OS.

The free space in the heap will not be returned to OS.

six。 The malloc memory in the program code has a corresponding free, so there won't be a memory leak?

In a narrow sense, memory leak refers to malloc memory, no free, resulting in memory waste until the end of the program. In a broad sense, memory leak means that the amount of memory used by the process is increasing, or greatly exceeds the upper limit of the original design of the system.

As mentioned in the previous section, free memory is not immediately returned to OS, and holes (fragments) in the heap are hard to really release unless they become the new top of the heap. So, as in case (5) of the previous example, 40k and 60k pieces of memory have been released, but if you need to apply for more than 60k (such as 70k) at this time, and there are no available fragments, you must apply to OS, and the actual memory used is still increased.

Therefore, with the frequent malloc and free of the system, especially for small blocks of memory, more and more unavailable fragments will be generated in the heap, resulting in "memory leaks". But this kind of "leak" phenomenon cannot be detected with valgrind.

The following figure shows the memory usage (RSS and VSZ) of MySQL when there are a large number of partition tables, suspected of a "memory leak".

Therefore, when we write programs, we cannot rely entirely on the implementation of glibc's malloc and free. A better way is to establish a memory pool that belongs to the process, that is, malloc large chunks of memory, small memory obtained from the memory pool, and free when the process ends or the block of memory is unavailable, which can greatly reduce fragmentation.

seven。 Since memory in the heap cannot be freed directly, why not use mmap to allocate it all?

Since the fragment in the heap cannot be released directly, it is mentioned in question 5 that the memory allocated by mmap can be free through munmap to achieve real release. Since the fragment in the heap cannot be released directly, resulting in a suspected "memory leak" problem, why doesn't malloc use mmap to implement it all? Use mmap only for large chunks of memory larger than 128k?

In fact, the interface sbrk/mmap/munmap for the process to apply for and release address space from OS is a system call, and frequent system calls consume system resources. Moreover, after the memory requested by mmap is munmap, reapplication will result in more page-missing interruptions. For example, using mmap to allocate 1m space, the first call produces a large number of page fault interrupts (1M/4K times). When 1m space is allocated again after munmap, a large number of page fault interrupts will occur again. Page fault interrupts are kernel behaviors that result in high CPU consumption in kernel mode. In addition, if you use mmap to allocate small memory, it will result in more fragmentation of the address space and a greater administrative burden on the kernel.

While the heap is a contiguous space, and the fragments in the heap have not been returned to OS, if the fragments are reusable, accessing the memory again is likely without any system calls and page-missing interruptions, which will greatly reduce the consumption of CPU.

Therefore, in the malloc implementation of glibc, the differences in behavior between sbrk and mmap as well as their advantages and disadvantages are fully taken into account. By default, large blocks of memory (128k) are allocated to obtain address space using mmap, and this critical value can also be modified through mallopt (M_MMAP_THRESHOLD,).

eight。 How do I view the missing page break information for a process?

You can view the missing page break information with the following command

Ps-o majflt,minflt-C ps-o majflt,minflt-p

Where majflt stands for major fault, which refers to big mistakes, and minflt stands for minor fault, which refers to minor mistakes. These two values represent the number of page-missing interruptions that have occurred since a process was started. The difference between majflt and minflt is that majflt means that you need to read and write to the disk. It may be that the corresponding memory pages need to be load to the physical memory in the disk, or there may be insufficient physical memory at this time, so you need to eliminate some physical pages to the disk.

For example, here is an example of mysqld.

Mysql@ TLOG_590_591:~ > ps-o majflt,minflt-C mysqldMAJFLT MINFLT144856 15296294

If the kernel state CPU of the process is used too much, one of the reasons may be the number of missing page interruptions per unit time, which can be checked by the above command.

If the MAJFLT is too large, it is likely to be out of memory.

If the MINFLT is too large, it is likely to allocate / release large chunks of memory frequently (128k), and malloc uses mmap to allocate it. In this case, the critical value can be increased by mallopt (M_MMAP_THRESHOLD,), or the memory pool can be implemented by the program.

nine。 How do I check the fragmentation of memory in the heap?

Glibc provides the following structures and interfaces to view heap memory and mmap usage.

Struct mallinfo {int arena; / * non-mmapped space allocated from system * / int ordblks; / * number of free chunks * / int smblks; / * number of fastbin blocks * / int hblks; / * number of mmapped regions * / int hblkhd; / * space in mmapped regions * / int usmblks; / * maximum total allocated space * / int fsmblks; / * space available in freed fastbin blocks * / int uordblks; / * total allocated space * / int fordblks; / * total free space * / int keepcost / * top-most, releasable (via malloc_trim) space * /}; / * return the memory usage of heap (main_arena) and * / struct mallinfo mallinfo () in mallinfo structure; / * output the usage of heap and mmap to stderr * / void malloc_stats ()

The following example can be used to verify the mallinfo and malloc_stats output.

# include # include size_t heap_malloc_total, heap_free_total, mmap_total, mmap_count;void print_info () {struct mallinfo mi = mallinfo (); printf ("count by itself:\ n") Printf ("\ theap_malloc_total=%lu heap_free_total=%lu heap_in_use=%lu\ n\\ tmmap_total=%lu mmap_count=%lu\ n", heap_malloc_total*1024, heap_free_total*1024, heap_malloc_total*1024-heap_free_total*1024, mmap_total*1024, mmap_count); printf ("count by mallinfo:\ n") Printf ("\ theap_malloc_total=%lu heap_free_total=%lu heap_in_use=%lu\ n\\ tmmap_total=%lu mmap_count=%lu\ n", mi.arena, mi.fordblks, mi.uordblks, mi.hblkhd, mi.hblks); printf ("from malloc_stats:\ n"); malloc_stats () } # define ARRAY_SIZE 200int main (int argc, char** argv) {char** ptr_ arr [array _ SIZE]; int i; for (I = 0; I < ARRAY_SIZE; iota +) {ptr_ arr [I] = malloc (I * 1024); if (I < 128c) heap_malloc_total + = I Else {mmap_total + = I; mmap_count++;}} print_info (); for (I = 0; I < ARRAY_SIZE; iTunes +) {if (I% 2 = = 0) continue Free (ptr_ arr [I]); if (I < 128) heap_free_total + = I; else {mmap_total-= I; mmap_count--;}} printf ("\ nafter free\ n") Print_info (); return 1;}

In this example, the first loop allocates memory blocks of index location (KB) size to each member of the pointer array, and counts the heap and mmap memory allocation by 128as the demarcation; the second loop is an odd subscript of the free index, while updating the count. The results obtained by the counting of the program and the mallinfo/malloc_stats interface are compared and printed to the terminal through print_info.

The following is an execution result:

Count by itself: heap_malloc_total=8323072 heap_free_total=0 heap_in_use=8323072 mmap_total=12054528 mmap_count=72count by mallinfo: heap_malloc_total=8327168 heap_free_total=2032 heap_in_use=8325136 mmap_total=12238848 mmap_count=72from malloc_stats:Arena 0:system bytes = 8327168in use bytes = 8325136Total (incl. Mmap): system bytes = 20566016in use bytes = 20563984max mmap regions = 72max mmap bytes = 12238848after freecount by itself: heap_malloc_total=8323072 heap_free_total=4194304 heap_in_use=4128768 mmap_total=6008832 mmap_count=36count by mallinfo: heap_malloc_total=8327168 heap_free_total=4197360 heap_in_use=4129808 mmap_total=6119424 mmap_count=36from malloc_stats:Arena 0:system bytes = 8327168in use bytes = 4129808Total (incl. Mmap): system bytes = 14446592in use bytes = 10249232max mmap regions = 72max mmap bytes = 12238848

As can be seen from the above, the program statistics are basically consistent with the information obtained by mallinfo, where heap_free_total represents the sum of memory fragments that have been released in the heap.

If you want to know how broken the chip in the heap is, you can get it through the fsmblks, smblks and ordblks in the mallinfo structure. These values represent the total number of fragments in different sizes, which are 080 bytes, 80 bytes 512 bytes, and 512 bytes 128k, respectively. If the values of fsmblks and smblks are too large, the fragmentation problem may be more serious.

However, there is a fatal problem with the mallinfo structure, that is, all its member definitions are int. In a 64-bit environment, the uordblks/fordblks/arena/usmblks in its structure can easily lead to overflow, which should be left over by history. Pay attention to it when using it!

ten。 Are there any other third-party implementations besides glibc's malloc/free?

In fact, many people begin to criticize the implementation of glibc memory management, that is, the problems of high concurrency, poor performance and memory fragmentation are more serious. Therefore, some third-party tools have emerged to replace the implementation of glibc, the most famous of which are google's tcmalloc and facebook's jemalloc.

This is the end of the content of "what is the Linux virtual memory management method". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.