Shulou(Shulou.com)06/01 Report--

This article mainly introduces the Linux system memory knowledge points, the article is very detailed, has a certain reference value, interested friends must read it!

First, walk into linux memory

1. What is memory?

1) memory, also known as main memory, is a storage space that can be directly addressed by CPU and is made of semiconductor devices.

2) memory is characterized by fast access speed.

2. The function of memory

1) temporarily store the operation data of cpu

2) data exchanged by external memory such as hard disk

3) ensure the stability and high performance of cpu computing

2. Linux memory address space

1. Linux memory address space Linux memory management panorama

2. Memory address-user mode & kernel state

User mode: Ring3 code that runs in user mode is subject to a lot of processor

Kernel state: Ring0 in the memory protection of the processor, kernel mentality

Three ways to switch from user mode to kernel mode: system call, exception, peripheral interrupt

Difference: each process has its own, independent, undisturbed memory space; user-mode programs cannot manipulate the kernel address space at will, which has a certain role of security protection; kernel-mode threads share kernel address space

3. Memory address-MMU address translation

MMU is a hardware circuit that consists of two components, one is a segmented component, and the other is a paging component.

The segmentation mechanism converts a logical address into a linear address.

The paging mechanism converts a linear address to a physical address

4. Memory address-segmentation mechanism

1) Segment selector

To facilitate quick retrieval of segment selectors, the processor provides six segment registers to cache segment selectors: cs,ss,ds,es,fs and gs

Base address of the segment (Base Address): the starting address of the segment in the linear address space

Limit of segments: the maximum offset that can be used within a segment in a virtual address space

2) segmented implementation

The value in the segment register of the logical address provides the segment descriptor, and then the segment base address and segment boundary are obtained from the segment descriptor, and then the linear address is obtained by adding the offset of the logical address.

5. Memory address-paging mechanism (32-bit)

The paging mechanism is done after the segmentation mechanism, which further translates linear addresses into physical addresses.

10-bit page directory, 10-bit page table item, 12-bit page offset address

The size of a single page is 4KB

6. User-mode address space

TEXT: code snippet executable code, string literals, read-only variables

DATA: data segment, global variable that has been initialized in the mapper

BSS section: stores uninitialized global variables in the program

HEAP: the run-time heap that uses the memory area requested by malloc during the run of the program

MMAP: the mapping area for shared libraries and anonymous files

STACK: user process stack

7. Kernel state address space

Direct mapping area: the maximum 896m interval starting from 3G in linear space, which is the direct memory mapping area.

Dynamic memory mapping area: this area is allocated by the kernel function vmalloc

Permanent memory mapping area: this area provides access to high-end memory

Fixed mapping area: there is only a 4k isolation zone at the top of this area and 4G, and each address entry serves a specific purpose, such as ACPI_BASE, etc.

8. Process memory space

Usually, user processes can only access the virtual address of user space, but not the virtual address of kernel space.

The kernel space is mapped by the kernel and will not change with the process; the kernel space address has its own corresponding page table, and the user process has different page tables.

3. Linux memory allocation algorithm

Memory management algorithm-a gift for people who hate to manage their own memory

1. Memory fragmentation

1) basic principles

Cause: the memory allocation is small, and the life cycle of these small memory allocation is longer, which will lead to memory fragmentation after repeated applications.

Advantages: improve allocation speed, facilitate memory management, and prevent memory leakage

Disadvantages: a large amount of memory fragmentation will make the system slow, low memory utilization and large waste.

2) how to avoid memory fragmentation

Use less dynamic memory allocation functions (use stack space as much as possible)

Allocate memory and free memory in the same function as much as possible

Try to apply for a large amount of memory at once instead of repeatedly applying for a small amount of memory.

Apply for the exponential power of 2 as much memory space as possible.

External fragment avoidance-partner system algorithm

Internal fragment avoidance-- slab algorithm

Manage the memory by yourself and design the memory pool

2. Partner system algorithm-organizational structure

1) concept

It provides an efficient allocation strategy for the kernel to allocate a set of consecutive pages, and effectively solves the problem of external fragmentation.

The allocated memory area is based on the page frame.

2) external fragments

External fragmentation refers to the memory free area that has not been allocated (does not belong to any process) but is too small to be allocated to a new process requesting memory space 3) organizational structure

All free pages are grouped into 11 block linked lists, and each block linked list contains page blocks with sizes of 1, 2, 4, 8, 16 and 32, 64128256512 and 1024 consecutive page frames, respectively. You can apply for up to 1024 consecutive pages, corresponding to continuous memory of 4MB size.

3. Partner system algorithm-application and recovery

1) Application algorithm

Apply for 2 ^ I page block storage space. If the 2 ^ I corresponding block linked list has free page blocks, it will be allocated to the application.

If there are no free page blocks, find out if there are any free page blocks in the block linked list corresponding to 2 ^ (I 1). If so, assign a 2 ^ I block linked list node to the application, and another 2 ^ I block linked list node is inserted into the 2 ^ I corresponding block linked list.

If there are no free page blocks in the 2 ^ (I 1) block linked list, repeat step 2 until the block linked list with free page blocks is found.

If still not, the return memory allocation failed

2) recovery algorithm

Free up 2 ^ I page block storage space, find the block linked list corresponding to 2 ^ I page blocks, whether there are page blocks that are contiguous with their physical addresses, and if not, there is no need to merge

If so, merge into page blocks of 2 ^ (I 1), and so on, continue to find next-level block links until they cannot be merged

3) conditions

Two blocks have the same size

Their physical addresses are contiguous

The page block is the same size.

4. How to allocate more than 4m memory?

1) Why limit the allocation of large chunks of memory

The more memory allocated, the more likely it is to fail.

There are few scenarios in which large blocks of memory are used.

2) the method of obtaining more than 4m memory in the kernel

Modify MAX_ORDER and recompile the kernel

Kernel boot selection passes "mem=" parameters, such as "mem=80M, to reserve some memory; then pass through the

Request_mem_region and ioremap_nocache map reserved memory to modules. Kernel startup parameters need to be modified and there is no need to recompile the kernel. However, this method does not support x86 architecture, only non-x86 architectures such as ARM and PowerPC are supported.

Calling the alloc_boot_mem function before the mem_init function in start_kernel pre-allocates a large chunk of memory, which requires the kernel to be recompiled

Vmalloc function, which is used by kernel code to allocate memory that is continuous in virtual memory but not necessarily contiguous in physical memory

5. Partner system-anti-fragmentation mechanism

1) immovable pages

These pages have a fixed location in memory and cannot be moved or recycled

Kernel code segment, data segment, memory from kernel kmalloc (), memory occupied by kernel thread, etc.

2) recyclable pages

These pages cannot be moved, but can be deleted. The kernel collects pages when they occupy too much memory or when there is a shortage of memory 3) removable pages

These pages can be moved at will, and the pages used by user-space applications fall into this category. They are mapped through the page table

When they are moved to a new location, the page table items are updated accordingly

6. Slab algorithm-- basic principle

1) basic concepts

The slab allocator used by Linux is based on an algorithm first introduced by Jeff Bonwick for the SunOS operating system.

Its basic idea is to put the frequently used objects in the kernel into the cache and keep them in the initial available state by the system. Such as process descriptors, this data is frequently requested and released in the kernel.

2) Internal fragments

The memory space that has been allocated is greater than the memory space required for the request. 3) basic goal

Reduce the internal fragmentation caused by the partner algorithm when allocating small blocks of continuous memory

Cache frequently used objects to reduce the time overhead of allocating, initializing, and releasing objects

Adjust objects through shading to make better use of hardware caching

7. The structure of slab allocator

Because objects are allocated and released from slab, a single slab can be moved between slab lists

Slab in the slabs_empty list is the main candidate for reaping

Slab also supports the initialization of generic objects, thus avoiding repeated initialization of an object for the same purpose.

8. Slab cache

1) normal cache

The allocation of small chunks of continuous memory provided by the slab allocator is achieved through the general cache.

The objects provided by the general cache have the size of a geometric distribution, ranging from 32 to 131072 bytes.

Two interfaces, kmalloc () and kfree (), are provided in the kernel to request and release memory, respectively.

2) dedicated cache

The kernel provides a complete set of interfaces for the application and release of dedicated caches, and allocates slab caches to specific objects according to the passed parameters.

Kmem_cache_create () is used to create a cache for a specified object. It allocates a cache descriptor to the new proprietary cache from the cache_cache normal cache and inserts this descriptor into the cache_chain linked list formed by the cache descriptor

Kmem_cache_alloc () allocates a slab in the cache specified by its parameter. Instead, kmem_cache_free () releases a slab in the cache specified by its parameter

9. Kernel state memory pool

1) basic principles

First apply for the allocation of a certain number of memory blocks of the same size (under normal circumstances) to be reserved for backup.

When there is a new memory requirement, part of the memory block is allocated from the memory pool, and if the memory block is not enough, continue to apply for new memory.

A significant advantage of this is that memory fragmentation is avoided as much as possible, thus improving the efficiency of memory allocation.

2) Kernel API

Mempool_create creates a memory pool object

The mempool_alloc allocation function gets the object

Mempool_free releases an object

Mempool_destroy destroys the memory pool

10. User-mode memory pool

1) C++ instance

11. DMA memory

1) what is DMA

Direct memory access is a hardware mechanism that allows the direct transfer of their Icano data between peripherals and main memory without the involvement of the system processor 2) the functions of the DMA controller

Can send system hold (HOLD) signal to CPU and request bus takeover

When CPU sends out the allow take-over signal, it is responsible for the control of the bus and enters the DMA mode.

The memory can be addressed and the address pointer can be modified to read and write to the memory.

Can determine the number of bytes of this DMA transfer, and determine whether the DMA transfer is over.

Send a DMA end signal to return the CPU to normal operation

2) DMA signal

DREQ:DMA request signal. It is the application signal that the peripheral puts forward the request to the DMA controller and the DMA operation.

DACK:DMA responds to the signal. It is the signal that the DMA controller indicates to the peripheral making the DMA request that the request has been received and is being processed.

The signal sent by the HRQ:DMA controller to the CPU to take over the request signal of the bus.

The signal sent by the HLDA:CPU to the DMA controller that allows you to take over the reply signal of the bus:

IV. Memory usage scenarios

Is the era of out of memory over? No, no matter how sufficient the memory is, you can't use it willfully.

1. Memory usage scenarios

Page management

Slab (kmalloc, memory pool)

User mode memory usage (malloc, relloc file mapping, shared memory)

Memory map of the program (stack, heap, code, data)

Data transfer between kernel and user mode (copy_from_user, copy_to_user)

Memory mapping (hardware registers, reserved memory)

DMA memory

2. Memory allocation function in user mode

Alloca requests memory from the stack, so there is no need to release

The memory space allocated by malloc has not been initialized, and programs that use the malloc () function will run normally at the beginning (memory space has not been reallocated), but problems may occur after a period of time (memory space has been reallocated)

Calloc initializes every bit in the allocated memory space to zero

Realloc expands existing memory space

A) if the current contiguous memory block is sufficient for realloc, just expand the space that p points to and return the pointer address of p. At this time, Q and p point to the same address

B) if the current continuous memory block is not long enough, find a place long enough, allocate a new piece of memory, Q, and copy the content pointed to by p to Q and return Q. And delete the memory space pointed to by p

3. Kernel state memory allocation function

Function allocation principle maximum memory other _ get_free_pages operate directly on the page frame 4MB is suitable for allocating a large amount of continuous physical memory kmem_cache_alloc based on slab mechanism 128KB is suitable for frequent applications to release blocks of the same size when using kmalloc to realize the most common allocation of 128KB based on kmem_cache_alloc Vmalloc can be used to establish the mapping from discontiguous physical memory to virtual address when memory is smaller than the size of the page frame, which is suitable for situations where large memory is needed, but where there is no requirement for address continuity, dma_alloc_coherent implements 4MB based on _ alloc_pages, which is suitable for DMA operation ioremap. Mapping from known physical address to virtual address is suitable for situations where the physical address is known, such as when the device driver alloc_bootmem starts kernel. Set aside a section of memory, the kernel can not be seen less than the physical memory size, memory management requirements are higher

4. Malloc applies for memory

When the malloc function is called, it looks for a block of memory large enough to satisfy the user's request along the free_chuck_list connection table.

The main job of the free_chuck_list join table is to maintain a free heap space buffer linked list

If the spatial buffer linked list does not find the corresponding node, the stack space of the process needs to be extended through the system call sys_brk.

5. Page missing exception

Apply for one or more physical pages through get_free_pages

Convert the pte address where addr is located in the process pdg map

Set the pte corresponding to addr as the first address of the physical page

System call: Brk- applies for memory less than or equal to 128kb memory-applies for memory greater than 128kb

6. Analysis of user process access memory

User-mode processes monopolize the virtual address space, and the virtual addresses of two processes can be the same

When accessing the user-mode virtual address space, if the physical address is not mapped, a page fault exception is issued through the system call

The page fault exception falls into the kernel, allocates physical address space, and establishes mapping with user-mode virtual addresses.

7. Shared memory

1) principle

It allows multiple unrelated processes to access the same part of logical memory

Data transfer between two running processes, shared memory will be a very efficient solution

Sharing data between two running processes is an efficient method of inter-process communication, which can effectively reduce the number of data copies.

2) shm interface

Shmget creates shared memory

Shmat initiates access to the shared memory and connects the shared memory to the address space of the current process

Shmdt separates shared memory from the current process

Fifth, memory use of those pits

1. C memory leak

New and delete functions are called without a match in the constructor and destructor of the class

Nested object pointers are not cleared correctly

The base class destructor is not defined as a virtual function

When the pointer of the base class points to the subclass object, if the destructor of the base class is not virtual, then the destructor of the subclass will not be called and the resources of the subclass will not be released correctly, resulting in a memory leak

Lack of copy constructor, pass by value will call (copy) constructor, reference pass will not call

An array of pointers to an object is not the same as an array of objects. A pointer to an object is stored in the array, which frees up not only the space of each object, but also the space of each pointer.

The lack of an overloaded assignment operator also copies the object member by member. If the size of the class is variable, the result is a memory leak.

2. C field pointer

The pointer variable is not initialized

The pointer is not set to NULL after being free or delete

Pointer operations go beyond the scope of variables. For example, pointers that return to stack memory are wild pointers.

Access to a null pointer (short judgment is required)

Sizeof cannot get the size of the array

An attempt was made to modify constants, such as: char p = "1234"; pairing 1'

3. C resource access conflict

Multithreaded shared variables are not decorated with valotile

Multithreaded access to global variables is not locked

Global variables are only valid for a single process

Multiple processes write shared memory data without synchronous processing

Mmap memory mapping, multi-process unsafe

4. STL iterator failure

The deleted iterator is invalid

Adding elements (insert/push_back, etc.) and deleting elements cause sequential container iterators to fail

Error example: delete the current iterator, the iterator will fail

Correct example: when iterator erase, you need to save the next iterator

5. C++ 11 Smart pointer

Replace auto_ptr with unique_ptr

Initialize a shared_ptr using make_shared

Weak_ptr intelligent pointer assistant (1) principle analysis:

(2) data structure:

(3) use method: a. Lock () to get the strong reference pointer b. Expired () of the managed object to detect whether the managed object has released c. Get () to access the smart pointer object.

C++ 11 is smaller, faster and safer.

Std::atomic atomic data type multithread safety

The cost of std::array fixed-length array is less than that of array, unlike std::vector, the length of array is fixed and cannot be dynamically expanded.

Std::vector vector slimming shrink_to_fit (): reduces capacity to the same size as size ()

Td::forward_list

Forward_list is a single linked list (std::list is a double linked list). When you only need to traverse sequentially, forward_list can save more memory, and the performance of insert and delete is better than that of list.

The time complexity of inserting, deleting and searching unordered containers implemented by std::unordered_map and std::unordered_set in hash is O (1). When the order of elements in the container is not concerned, containers using unordered can achieve higher performance. 6. How to view memory

Memory usage in the system: / proc/meminfo

Memory usage of the process: / proc/28040/status

Total memory usage for query: free

Query process cpu and memory usage ratio: top

Virtual memory statistics: vmstat

Memory consumption ratio and sorting of processes: ps aux-sort-rss

Free system memory cache: / proc/sys/vm/drop_caches

These are all the contents of the article "what are the knowledge points in the Linux system?" Thank you for your reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.