Shulou(Shulou.com)06/02 Report--

Editor to share with you a case study of memory management in Linux, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Under linux, when you use commands such as top,vmstat,free to check the memory usage of a system or process, you often see buff/cache memeory,swap,avail Mem and so on. What do they mean? This article will talk about memory management under Linux and answer this question in the future.

Discussing the memory management under Linux is actually discussing the implementation of virtual memory under Linux. I am not a kernel expert, so this article will only introduce some conceptual things, will not go into the implementation details, and may not be described accurately in some places.

In the early days, physical memory was limited, and people wanted programs to use more memory space than actual physical memory, so the concept of virtual memory appeared, but with the passage of time, the meaning of virtual memory has gone far beyond the original idea.

1. Virtual memory

Virtual memory is a technology used by Linux to manage memory. It makes each application think that it has an independent and continuous available memory space (a continuous and complete address space), when in fact, it is usually mapped to multiple physical memory segments. some are temporarily stored on external disk storage and loaded into memory when needed.

The size of the virtual address that each process can use is related to the number of CPU bits. On a 32-bit system, the virtual address space is 4G, on a 64-bit system, it is 2 ^ 64 =? It doesn't count anymore. The actual physical memory may be much smaller than the size of the virtual address space.

The virtual address is closely related to the process, and the same virtual address in different processes may not necessarily point to the same physical address, so there is no point in leaving the process to talk about the virtual address.

Note: many online articles equate virtual memory with swap space, but the description is not rigorous enough. Swap space is only part of the big blueprint of virtual memory.

The relationship between virtual memory and physical memory

The following table shows the relationship between them very intuitively.

Process X process yawry + +-+ | VPFN7 |-- + | | VPFN7 | +-+ | process Y of process X | VPFN6 | | Page Table Page Table +-| VPFN6 | +-+ | +-+ | +-+ | +-+ | VPFN5 | +-> |.... |-- + +-|. |. |-- +-+ | PFN4 |. | VPFN4 | +-+ | +-+ | | +-+ | +-+ | | +-+ | VPFN3 |-- + | |. | +-- > | PFN3 | | PFN0 | | VPFN0 | +-+ virtual memory physical memory | Virtual memory PFN (the page frame number): page number

When a process executes a program, it needs to read the instructions of the process from memory first, and then execute them. The virtual address is used to obtain the instructions. This address is determined when the program is linked (the kernel adjusts the address range of the dynamic library when the kernel loads and initializes the process). In order to obtain the actual data, CPU needs to convert the virtual address into a physical address, and CPU needs to use the page table of the process when translating the address. The data in page table is maintained by the operating system.

Note: Linux kernel code accesses memory using actual physical addresses, so there is no translation from virtual addresses to physical addresses, only application layer programs need it.

In order to facilitate conversion, Linux splits virtual memory and physical memory into fixed-size pages. X86 systems generally have a memory page size of 4K, and each page is assigned a unique number, which is the page number (PFN).

As you can see from the figure above, the page of virtual memory and physical memory is mapped by page table. The virtual memory of processes X and Y is independent of each other, and the page table is also independent, sharing physical memory between them. Processes can access their own virtual address space at will, while page table and physical memory are maintained by the kernel. When a process needs to access memory, CPU translates the virtual address into a physical address according to the process's page table, and then accesses it.

Note: not every page in the virtual address space is associated with a corresponding Page Table. Only after the virtual address is assigned to the process, that is, after the process calls a similar malloc function, the system will add a record in the Page Table for the corresponding virtual address. If the process accesses a virtual address that is not associated with Page Table, the system will throw a SIGSEGV signal, causing the process to exit. This is why segmentfault often appears when we access wild pointers. In other words, although each process has a 4G (32-bit system) virtual address space, only those that have been requested by the system can be used, and accessing the unallocated address space will result in a segmentfault error. Linux will not map the virtual address 0 anywhere, so we must report a segmentfault error when we access the null pointer.

Third, the advantages of virtual memory

● has a larger address space and is contiguous, making programming and linking easier.

● process isolation: there is no relationship between the virtual addresses of different processes, so the operation of one process will not affect other processes

● data protection: each virtual memory has corresponding read and write attributes, which can protect the code segment of the program from being modified, the data block cannot be executed, etc., and increase the security of the system.

● memory mapping: after having virtual memory, you can map files on disk (executable files or dynamic libraries) directly to virtual address space, thus delaying the allocation of physical memory. Only when you need to read the corresponding file, can it really be loaded from disk into memory, and when memory is tight, this part of memory can be cleared out to improve the efficiency of physical memory utilization. And all of this is transparent to the application.

● shared memory: dynamic libraries, for example, just store one copy in memory and map it to the virtual address space of different processes, making the process feel that it has exclusive ownership of the file. Memory sharing between processes can also be achieved by mapping the same piece of physical memory to different virtual address spaces of processes.

● physical memory management: the physical address space is all managed by the operating system, and the process cannot be allocated and recycled directly, so that the system can make better use of memory and balance the memory requirements between processes.

● other: with virtual address space, swap space and COW (copy on write) and other functions can be easily implemented.

IV. Page table

Page table can be simply understood as a linked list of memory mapping (of course, the actual structure is very complex), in which each memory mapping maps a virtual address to a specific resource (physical memory or external storage space). Each process has its own page table, which has nothing to do with the page table of other processes.

5. Memory mapping

Each memory mapping is a description of a virtual memory, including the starting location, length, permissions (such as whether the data in this memory can be read, written, and executed), and associated resources (such as page on the physical memory page,swap space, file contents on disk, etc.).

When the process requests memory, the system will return the virtual memory address, create a memory mapping for the corresponding virtual memory and put it into the page table, but at this time the system will not necessarily allocate the corresponding physical memory, the system will generally allocate the physical memory and associate it to the corresponding memory mapping only when the process actually accesses this memory, which is called delay allocation / on-demand allocation.

Each memory mapping has a tag to indicate the associated physical resource type, which is generally divided into two categories, namely anonymous and file backed. Among these two categories, there are some subcategories, such as more specific shared and copy on write types under anonymous, and more specific device backed types under file backed. Here is what each type represents:

File backed

This type means that the physical resources corresponding to memory mapping are stored in a file on disk, and the information it contains includes file location, offset, rwx permissions, and so on.

When the process accesses the corresponding virtual page for the first time, because the corresponding physical memory is not found in the memory mapping, the CPU will report the page fault interrupt, and then the operating system will handle the interrupt and load the contents of the file into the physical memory, and then update the memory mapping so that the CPU can access the virtual address next time. Data loaded into memory in this way is usually put into page cache, which will be described later on page cache.

General program executable files and dynamic libraries are mapped to the virtual address space of the process in this way.

Device backed

Similar to file backed, except that the back end is mapped to the physical address of the disk, for example, when physical memory is swap out, it will be marked as device backed.

Anonymous

The data segment and stack space used by the program itself, as well as the shared memory allocated through mmap, cannot find the corresponding file on disk, so this part of the memory page is called anonymous page. The biggest difference between anonymous page and file backed is that when memory is tight, the system will directly delete the physical memory corresponding to file backed, because the next time it is needed, it can be loaded into memory from disk, but anonymous page cannot be deleted and can only be swap out.

Shared

Multiple memory mapping in the Page Table of different processes can be mapped to the same physical address, and the same content can be accessed through the virtual address (the virtual address may be different in different processes). When the contents of memory are modified in one process, it can be immediately read in another process. This method is generally used to achieve high-speed inter-process data sharing (such as mmap). When the memory mapping marked shared is deleted and recycled, the reference count on the physical page needs to be updated so that the physical page count is changed to 0 and then recycled.

Copy on write

Copy on write is based on shared technology, when reading this type of memory, the system does not need to do any special operation, but when writing this memory, the system will generate a new memory and copy the data in the original memory to the new memory, then associate the new memory to the corresponding memory mapping, and then perform the write operation. Many functions in Linux rely on copy on write technology to improve performance, such as fork and so on.

Through the above introduction, we can simply summarize the memory usage process as follows:

1. The process sends a memory request to the system

2. The system will check whether the virtual address space of the process is used up, and if there is any left, assign the virtual address to the process.

3. The system creates a corresponding memory mapping (possibly multiple) for this virtual address and puts it into the page table of the process

4. The system returns the virtual address to the process, and the process begins to access the virtual address

5. CPU finds the corresponding memory mapping in the page table of the process according to the virtual address, but the mapping is not associated with physical memory, so a missing page interrupt occurs.

6. After the operating system receives the page fault interrupt, it allocates the real physical memory and associates it to the corresponding memory mapping

7. After the interrupt processing is completed, CPU can access the memory

Of course, page fault interrupts do not occur every time, but only when the system feels it is necessary to delay the allocation of memory, that is, in step 3 above, the system allocates real physical memory and is associated with memory mapping.

VI. Other concepts

As long as the operating system implements the mapping between virtual memory and physical memory, it will work properly, but there are many things to consider to make memory access more efficient. Here we can look at some other concepts related to memory and their role.

MMU (Memory Management Unit)

MMU is a module of CPU that converts the virtual address of a process into a physical address. To put it simply, the input of this module is the page table and virtual address of the process, and the output is the physical address. The speed of translating virtual addresses into physical addresses directly affects the speed of the system, so CPU includes this module for acceleration.

TLB (Translation Lookaside Buffer)

As mentioned above, the input of MMU is page table, and the page table is stored in memory. Compared with the cache of CPU, the speed of memory is very slow, so in order to further speed up the translation speed from virtual address to physical address, Linux invented TLB, which exists in the L1 cache of CPU and is used to cache the mapping of found virtual address to physical address, so check TLB before the next translation. If it is already inside, there is no need to call MMU.

Allocate physical pages on demand

Because physical memory is much less than virtual memory in reality, the operating system must allocate physical memory very carefully to maximize memory utilization. One way to save physical memory is to load only the data corresponding to the virtual page that is currently in use into memory. For example, for a large database program, if you only use the query operation, then the code snippets responsible for inserting and deleting and other parts of the code do not need to be loaded into memory, thus saving a lot of physical memory. This method is called physical memory page allocation on demand, also known as deferred loading.

The implementation principle is very simple, that is, when CPU accesses a virtual memory page, if the data corresponding to the virtual memory page has not been loaded into physical memory, CPU will notify the operating system that page fault has occurred, and then the operating system is responsible for loading the data into physical memory. Because loading data into memory is time-consuming, CPU does not wait there, but dispatches other processes, and the next time it is dispatched to that process, the data is already in physical memory.

Linux mainly uses this way to load executable files and dynamic libraries. When the program is scheduled by the kernel, the kernel maps the executable files and dynamic libraries of the process to the virtual address space of the process, and loads only a small part of the data to be used in physical memory, and the rest is loaded only when CPU accesses them.

Swap space

When a process needs to load data into physical memory, but the actual physical memory has been used up, the operating system needs to reclaim some page from physical memory to meet the needs of the current process.

For file backed memory data, that is, the data in physical memory comes from files on disk, the kernel removes this part of data directly from memory to release more memory, and the next time a process needs to access this part of data, it is loaded into memory from disk. However, if this part of the data is modified and not written to the file, then this part of the data becomes dirty data, and the dirty data can not be deleted directly, but can only be moved to the exchange space. (executable files and dynamic library files will not be modified, but disk files mapped to memory through mmap+private may be modified. The mapped memory in this way is special. It is file backed before modification, and becomes anonymous after modification but before being written back to disk.)

For anonymous memory data, there is no corresponding file on disk, this part of data can not be deleted directly, but moved to the swap space by the system. Swap space is a special space reserved on disk, which is used by the system to store the data that is not often accessed in memory. The next time a process needs to access the data on the swap space, the system loads the data into memory. Because the swap space is on disk, the access speed is much slower than memory, and frequent read and write swap space can cause performance problems.

For a detailed introduction of swap space, please refer to Linux Exchange Space.

Shared memory

With virtual memory, shared memory between processes becomes particularly convenient. All memory access for processes is achieved through virtual addresses, and each process has its own page tables. When two processes share a piece of physical memory, simply map the page number of the physical memory to the page table of the two processes, so that the two processes can access the same piece of physical memory through different virtual addresses.

As you can see from the figure above, process X and process Y share the physical memory page PFN3. In process X, PFN3 is mapped to VPFN3, while in process Y, PFN3 is mapped to VPFN1, but the physical memory accessed by the two processes through different virtual addresses is the same block.

access control

Each virtual memory to physical memory mapping record (memory mapping) in page table contains a control information that can be used to check whether the current operation is legitimate when the process wants to access a piece of virtual memory.

Why do you need to do this test? For example, some memory contains executable code of the program, so it should not be modified; some memory stores data used when the program is running, then this part of memory can only be read and written, and should not be executed; some memory stores kernel code, so it should not be executed in user mode; with these checks, the security of the system will be greatly enhanced.

Huge pages

Because the cache of CPU is limited, the data cached in TLB is also limited, and after using huge page, due to the larger memory per page (for example, from 4K to 4m), although the number of records in TLB remains the same, the address space that these records can cover becomes larger, which is equivalent to the larger mapping range that can be cached in TLB of the same size, thus reducing the number of calls to MMU. Speed up the translation of virtual addresses to physical addresses.

Caches

To improve system performance, Linux uses some cache related to memory management and tries to use free memory for these cache. These cache are shared globally by the system:

Buffer Cache

It is used to buffer the data on the block device, such as the disk. When reading and writing the block device, the system will store the corresponding data in this cache, and the next time it is accessed, it can directly take the data from the cache, thus improving the efficiency of the system. The data structure in it is a mapping of block device ID and block numbers to specific data. As long as the block device ID and block numbers are used, the corresponding data can be found.

Page Cache

This cache is mainly used to speed up the reading and writing of files on disk. The data structure in it is the mapping of the file ID and offset to the file content, and the corresponding data can be found according to the file ID and offset (here the file ID may be inode or path, I did not study it carefully).

As you can see from the above definition, page cache and buffer cache overlap, but the reality is that buffer cache caches only those parts of the content that page cache does not cache, such as the metadata of files on disk. So in general, compared with page cache, the size of Buffer Cache is basically negligible.

Of course, there are some disadvantages to using cache, for example, it takes time and space to maintain cache,cache, and if something goes wrong, the whole system will fail.

VII. Summary

With the knowledge described above, let's take a look at the question we just asked, taking the output of the top command as an example:

KiB Mem: 500192 total, 349264 free, 36328 used, 114600 buff/cacheKiB Swap: 524284 total, 524284 free, 0 used. 433732 avail Mem

KiB Mem stands for physical memory, KiB Swap stands for swap space, and their units are KiB.

Total, used, and free have nothing to say, just how much is in total, how much is then used, and how much is left.

Buff/cached represents the total amount of buff and cache used, and buff represents how much space buffer cache takes up. Because it is mainly used to cache the metadata of files on disk, it is generally small and negligible compared with cache; cache represents the sum of page cache and other relatively small and fixed cache. Basically, cache is about the exact value of page cache,page cache. You can get it by viewing the Cached in / proc/meminf. Because page cache is used to cache the contents of files on disk, so it takes up a lot of space, Linux generally uses as much free physical memory as possible for page cache.

Avail Mem represents the amount of physical memory that can be allocated by the process next time, which is generally larger than free, because the system can immediately free up some space in addition to free space.

So how can you tell if there is an exception in the current memory usage? There are the following points for reference:

The value of ● Mem free is small, and the value of buff/cache is also small.

A small value of free does not necessarily mean that there is a problem, because Linux will use memory for page cache as much as possible, but if the value of buff/cache is also small, it means that memory is tight and the system does not have enough memory for cache. If the current server deployment is an application that needs to read and write disks frequently, such as FTP server, then the impact on performance will be very great.

The value of ● Swap used is relatively large

This situation is more serious than the above. Normally, swap should be rarely used. A large use value means that more swap space is used. If you see swap in/out more frequently through the vmstat command, it means that the system is seriously out of memory and the overall performance has been seriously affected.

The above is all the contents of the case study of memory management in Linux. Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.