Shulou(Shulou.com)06/03 Report--

This article mainly introduces "the memory management mechanism and tuning mode of Python development". In the daily operation, I believe that many people have doubts about the memory management mechanism and tuning mode of Python development. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "memory management mechanism and tuning mode of Python development". Next, please follow the editor to study!

Memory management mechanism: reference counting, garbage collection, memory pool.

Reference counting: reference counting is a very efficient means of memory management. When a Python object is referenced, its reference count increases by 1, and when it is no longer referenced by a variable, the reference count is subtracted by 1. The object is deleted when the reference count is equal to 0.

3. Garbage collection:

1. Reference count

Reference counting is also a garbage collection mechanism, and it is also one of the most intuitive and simplest garbage collection techniques. When the reference count of an object in Python drops to 0, it means that there are no references to the object, and the object becomes garbage to be recycled. For example, a new object is assigned to a reference, and the reference count of the object becomes 1. If the reference is deleted and the reference count of the object is 0, the object can be garbage collected. However, if there are circular references, the reference counting mechanism is no longer effective.

two。 Mark clear

If both objects have a reference count of 1, but only circular references exist between them, then both objects need to be recycled, that is, although their reference count is non-zero, in fact, the valid reference count is 0. So remove the circular reference first, and you will get a valid count of these two objects.

3. Generation by generation recovery

Judging from the previous garbage collection mechanism such as "mark-clear", the additional operations caused by this garbage collection mechanism are actually related to the total number of memory blocks in the system. The more memory blocks need to be reclaimed, the more additional operations garbage detection brings, and the less additional operations garbage collection brings. Conversely, when there are fewer blocks of memory to be reclaimed, garbage detection will bring fewer additional operations than garbage collection.

For example:

When some memory block M is still alive after three garbage collection cleanings, we divide the memory block M into a set A, and the newly allocated memory is divided into set B. When garbage collection starts to work, in most cases, only collection B is garbage collected, while collection An is garbage collected after a long time, which makes the garbage collection mechanism need less memory to deal with. Efficiency is naturally improved. In this process, some memory blocks in set B will be transferred to set A because of their long survival time. Of course, there is actually some garbage in set A, and the collection of these garbage will be delayed because of this generational mechanism.

Memory pool:

The memory mechanism of Python is in the shape of a pyramid. Layer 2 of Python is mainly operated by an operating system.

Layer 0 is operated by memory allocation and release functions such as malloc,free in C

Layers 1 and 2 are memory pools, which are implemented by the interface function PyMem_Malloc of Python. When the object is less than 256K, this layer allocates memory directly.

Layer 3 is the top layer, that is, our direct operation on Python objects; Python will perform a large number of malloc and free operations during operation, frequently switching between user mode and core state, which will seriously affect the efficiency of Python execution.

In order to accelerate the execution efficiency of Python, Python introduces a memory pool mechanism to manage the allocation and release of small chunks of memory.

The default demarcation point between small chunks of memory and large chunks of memory in Python is set at 256bytes. When the requested memory is less than 256bytes, PyObject_Malloc will apply for memory in the memory pool; when the requested memory is larger than 256bytes, the behavior of PyObject_Malloc will be reduced to the behavior of malloc. Of course, by modifying the Python source code, we can change this default value, thus changing the default memory management behavior of Python.

Understand the means of python tuning

1. Manual garbage collection

two。 Raise the garbage collection threshold

3. Avoid circular references (manually unloop references and use weak references)

At this point, the study of "memory management mechanism and tuning mode of Python development" is over. I hope to be able to solve everyone's doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.