Shulou(Shulou.com)05/31 Report--

This article mainly introduces the Linux memory management memory recovery example analysis, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

1.1 goal of memory recovery

Not all physical memory can be recycled. For example, if the memory of kernel code segments is recycled, the system will not be able to run normally. Generally, kernel code segments, data segments, memory from kernel kmalloc (), memory occupied by kernel threads, etc., cannot be recycled. Other memory is the goal we want to recover.

The reclaimed memory is mainly composed of the memory occupied by the user-mode process and some of the memory used by the kernel itself at run time. The memory occupied by user-mode processes is mainly our common process code segments, data segments, stacks, etc., the memory used by the kernel to run is mainly disk cache (such as index node, directory item cache), page cache (cache generated by the system when accessing files), physical memory used by the named mapping used by mmap () files. Although the latter internal talents are also the memory used by the kernel management, when they are reclaimed, it will at most affect the performance of the kernel and will not cause the system to fail to run.

1.2 timing of memory recovery

1. Memory shortage recovery: grow_buffers () cannot get the buffer page, alloc_page_buffers () cannot get the page temporary buffer header, and _ _ alloc_pages () can no longer allocate a set of consecutive page frames for the given memory area.

2. Cycle recovery: if necessary, activate the corresponding kernel thread to execute the memory recovery algorithm: kswapd () kernel thread to check whether the number of free page boxes in a memory management area is lower than the elevation of the pages_ high value. The events kernel thread, a worker thread, reclaims all idle slab in the cache.

1.3 memory recovery strategy

1.3.1 Classification of memory recovery

The main purpose of memory recycling is to reclaim two types of memory: the least recently used memory and the free slab in the memory cache. The former mainly includes the code segment, data segment, stack, file mapping memory and page high-speed memory of the user-mode process, while the latter mainly includes disk cache and some other free memory cache.

Recently, at least memory is stored on a lru linked list. Each memory management area zone has a lru structure, which contains two link headers, active and inactive. The active linked list records the current active messages, and the inactive is used to record the currently inactive messages. Usually we look back on the memory page on the inactive linked list on lru. At the same time, in the process of memory collection, the corresponding least recently used memory pages are added from the active linked list to the inactive linked list. The kernel data structure page of each memory page has a tag bit PG_referenced that doubles the time it takes for a page to move from "inactive" to "active" and vice versa. For example, a page may not be reacted within an hour. You can't think that it is active just because of an occasional visit. It takes two times to think that it is an active page. The following is a diagram of the changes of the page transferred on the inactive and active linked lists.

There are often some completely idle slab in the Slab memory cache, which is another goal for our recycling.

1.3.2 reverse mapping

Physical pages that can be accessed directly through the user-state linear address space can be divided into anonymous pages and file mapping pages. Anonymous pages refer to physical pages that do not correspond to specific files, such as code snippets, stacks, and so on. Mapping pages refer to physical pages mapped to a certain part of the file, and mmap () is usually used for related mapping.

For anonymous mapping and file mapping, a piece of physical memory may be used in the page tables of multiple processes, for example, for anonymous mapping, fork () a process will initially share the physical memory of the parent process, and for file mapping, multiple processes may map to the same part of the file at the same time. So when the page is recycled, the page needs to be removed from all page table references. This method is called reverse mapping. To find the page table items that use these physical pages, you need to find the page table that references them, and the address of the page table is recorded in the memory descriptor of each process, and each vm_area_struct used to describe the user-state address space of the process records a pointer to the memory descriptor to which it belongs. So as long as you find the vm_area_struct that references them through the physical page, you can find the memory descriptor, find the page table, and find the corresponding page table item.

Reverse mapping of anonymous pages:

For anonymous pages, the mapping field of each page points to an anon_vma descriptor, and there is a link header in the anon_vma descriptor, where all vm_area_struct referencing the page is stored. Page,anon_vma,vm_area_struct the relationship between these data structures is shown in the following figure:

For anonymous pages, they are referenced by other address spaces, basically because when the fork () process, the child process copies the address space of the parent process, thus being referenced. The process for each vm_area_struct to join the linked list of anon_vma is as follows:

Suppose there is a current process p, and then fork gives a child process c.

1. When process P adds * * physical pages to a vm_area_struct, such as a page fault exception, dynamically allocates an anon_vma data structure, adds vm_area_struct to the linked list managed by the anon_vma, the anon_vma field in the vm_area_struct structure points to the anon_vma, and assigns the mapping field in the page to the anon_vma. For subsequent physical pages that apply for the vm_area_struct, the mapping field is assigned to the anon_vma.

2. When the process p executes fork (), in the process of fork, dup_mmap () will be called to copy the linear address space of process p. In dup_mmap (), each vm_area_struct of process p will be copied, added to its own address space, and vm_area_struct will be added to the linked list managed by anon_vma, see anon_vma_link (). At this time, the page applied for for process p is shared by process c, the anon_vma can be found through the mapping field of the page, and the process pforce c can be traversed from anon_vma.

3. Consider a problem: the requested memory page with a page fault exception is triggered in process c, and its mmapping is assigned to the vma_anon of the vm_area_struct to which it belongs, but process p does not use the page, so the vm_area_struct under the vma_anon pointed to by the mapping field of a physical page may not contain the physical page.

Reverse mapping of the file mapping page:

For each file mapping page, the page mapping field points to the address_space data structure of the corresponding file, and there is a struct prio_tree_root i_mmap field in the address_space that points to a priority tree, which organizes all the vm_area_struct that map the contents of the file. In this tree, the node base address and heap address of the tree are the start address and end address of the mapped file content, respectively. If multiple processes map the address segment at the same time, the vm_area_struct will be concatenated on the node with a linked list.

1.3.3 introduction to memory recovery process

We don't pay attention to sleep recovery. We mainly introduce memory shortage recovery and cycle recovery:

1, memory shortage recovery the main function is try_to_free_pages (), this function will execute a loop, according to the priority from 12 to 0, then call shrink_caches (), shrink_slab () to recycle pages until at least 32 memory pages are recycled.

Call the following helper functions in turn:

Shrink_caches (): call shrink_zone () to recycle the pages above the lru for each zone in the incoming zone linked list.

Shrink_slab (): reclaim the disk cache such as disk index node cache and directory item index node. Since both disk index node and directory item index node are allocated from the slab cache, this will lead to the generation of free slab, and the idle slab will then be recycled in the periodically recycled cache_reap work queue. It is probably because the free slab will eventually be cleared, so that such a function name will be given. ^ _ ^

Shrink_zone (): recycle the inactive pages in the lru linked list on the memory management area. When the inactive pages are insufficient, call refill_inactive_zone () to supplement the inactive pages in the inactive linked list on lru, while shrink_zone () calls shrink_cache () to collect the pages. The specific parsing of this function can refer to the following source code analysis.

Shrink_list (): this helper function is called in shrink_cache (), which traverses the inactive page list passed in shrink_cache () and collects each page. The specific parsing of this function can refer to the source code parsing below.

Refill_inactie_zone (): this auxiliary function moves the active pages on the lru active linked list to the inactive linked list according to certain rules to supplement the recyclable pages. There are two types of pages in the lru linked list, one is the page belonging to the user state space, such as the code segment and data segment of the user state process, and the other is the page in the page cache, in order to reduce the impact on the application. Priority will be given to the collection of page cache pages, and user-mode process pages will be recycled appropriately for the overall performance of the system. Choose according to the following empirical formula:

Exchange tendency value = mapping ratio / 2 + load value + exchange value

2. The kswapd process usually sleeps in the system, but when _ _ alloc_page () finds that the remaining pages in each management area are lower than the warning value (calculated by the pages_low field and protection field of the memory management descriptor), the kswapd process will be activated for page recycling until the recycled pages make the remaining pages in the management area higher than zone- > pages_high. In essence, shrink_zone () and shrink_slab () are also called.

3. The cache_reap work queue runs periodically to recycle pages occupied by idle slab in the slab cache.

1.4 Analysis of related source code

Static void shrink_zone (struct zone * zone, struct scan_control * sc) {unsigned long nr_active; unsigned long nr_inactive; / / get the number of pages that can be scanned according to priority. The higher the priority is, the less urgent the page is, and the least number of pages that can be scanned is zone- > nr_scan_active + = (zone- > nr_active > > sc- > priority) + 1; nr_active = zone- > nr_scan_active If (nr_active > = SWAP_CLUSTER_MAX) zone- > nr_scan_active = 0; else nr_active = 0; zone- > nr_scan_inactive + = (zone- > nr_inactive > > sc- > priority) + 1; nr_inactive = zone- > nr_scan_inactive / / if there are few inactive pages, you can ignore the past and record the skipped pages in nr_scan_inactive / / leave them to be processed next time if (nr_inactive > = SWAP_CLUSTER_MAX) zone- > nr_scan_inactive = 0; else nr_inactive = 0; / / set the number of pages to be recycled as 32 sc- > nr_to_reclaim = SWAP_CLUSTER_MAX / / start recycling pages, scan 32 pages at a time, too many do not do Oh! While (nr_active | | nr_inactive) {if (nr_active) {/ / set the number of inactive pages to be scanned each time, which need to be put into inactive list sc- > nr_to_scan = min (nr_active, (unsigned long) SWAP_CLUSTER_MAX); nr_active-= sc- > nr_to_scan; / / Page refill_inactive_zone (zone, sc) in supplementary inactive list } if (nr_inactive) {/ / set the pages to be scanned each time, up to 32 pages sc- > nr_to_scan = min (nr_inactive, (unsigned long) SWAP_CLUSTER_MAX); nr_inactive-= sc- > nr_to_scan; / / start to formally recycle the page shrink_cache (zone, sc) in inactive list; / / 32 pages have been recycled, and you're done! If (sc- > nr_to_reclaim lru); / / page is locked and cannot recycle if (TestSetPageLocked (page)) / / page is locked? Goto keep; BUG_ON (PageActive (page)); / / page is being writeback and cannot recycle if (PageWriteback (page)) / / page is writeback? Goto keep_locked; sc- > nr_scanned++; / * Double the slab pressure for mapped and swapcache pages * / if (page_mapped (page) | | PageSwapCache (page)) sc- > nr_scanned++; / / check whether the page has been visited recently referenced = page_referenced (page, 1, sc- > priority gfp_mask & _ _ GFP_FS) | (PageSwapCache (page) & & (sc- > gfp_mask & _ GFP_IO)) / / this page is mapped to a user page table if (page_mapped (page) & & mapping) {/ / clear all page table entries in the user page table switch (try_to_unmap (page)) {case SWAP_FAIL: goto activate_locked; case SWAP_AGAIN: goto keep_locked; case SWAP_SUCCESS: / * try to free the page below * /}} / / the page is dirty. Haha, prepare to write the hard drive to the file or swapcache. If (PageDirty (page)) {if (referenced) goto keep_locked; if (! may_enter_fs) goto keep_locked; if (laptop_mode & &! sc- > may_writepage) goto keep_locked / * Page is dirty, try to write it out here * / / write pages to disk switch (pageout (page, mapping)) {case PAGE_KEEP: goto keep_locked; case PAGE_ACTIVATE: goto activate_locked; case PAGE_SUCCESS: if (PageWriteback (page) | | PageDirty (page)) goto keep; if (TestSetPageLocked (page)) goto keep; if (PageDirty (page) | | PageWriteback (page)) goto keep_locked; mapping = page_mapping (page); case PAGE_CLEAN: / * try to free the page below * /}} / / if the page is a buffer page, release the corresponding buffer_head to if (PagePrivate (page)) {if (! try_to_release_page (page, sc- > gfp_mask)) goto activate_locked; if (! mapping & & page_count (page) = = 1) goto free_it;} if (! mapping) goto keep_locked / * truncate got there first * / spin_lock_irq (& mapping- > tree_lock); / / if the page is dirty or the reference count of page is 2, both if (page_count (page)! = 2 | | PageDirty (page)) {spin_unlock_irq (& mapping- > tree_lock); goto keep_locked } # ifdef CONFIG_SWAP / / arrives here, indicating that the page is only shared by swap cache or page cache and / / fpra, and needs to be deleted from swap cache or page cache. If (PageSwapCache (page)) {swp_entry_t swap = {.val = page- > private}; / / remove _ _ delete_from_swap_cache (page) from swap cache; spin_unlock_irq (& mapping- > tree_lock); swap_free (swap); _ _ put_page (page); / * The pagecache ref * / goto free_it } # endif / * CONFIG_SWAP * / / remove the page from the page cache _ _ remove_from_page_cache (page); spin_unlock_irq (& mapping- > tree_lock); _ _ put_page (page); free_it: unlock_page (page); reclaimed++; if (! pagevec_add (& freed_pvec, page)) _ pagevec_release_nonlru (& freed_pvec); continue Activate_locked: / / set the page to active page, and then put it in lru's active linked list SetPageActive (page); pgactivate++; keep_locked: / / keep the status of the page unchanged and put unlock_page (page) in the corresponding lru active or inactive linked list; keep: / / put the unrecyclable page into the ret_pages linked list list_add (& page- > lru, & ret_pages); BUG_ON (PageLRU (page)) } / / put the pages that cannot be recycled into page_list, and after the function returns, go to deal with list_splice (& ret_pages, page_list); / / all the pages that can be released here will be released, and ^ _ ^ if (pagevec_count (& freed_pvec)) _ pagevec_release_nonlru (& freed_pvec); mod_page_state (pgactivate, pgactivate); sc- > nr_reclaimed + = reclaimed; return reclaimed } Thank you for reading this article carefully. I hope the article "sample Analysis of Linux memory Management Recycling" shared by the editor will be helpful to you. At the same time, I also hope that you will support and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.