Shulou(Shulou.com)06/01 Report--

This article mainly explains the "Python C extension citation counting problem analysis", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "Python C extension citation counting problem analysis"!

Python GC mechanism

For a high-level language like Python, developers do not need to manage and maintain memory on their own. Python adopts garbage collection mechanism, which is mainly based on reference counting mechanism and supplemented by tag-clear mechanism and generational collection mechanism.

First, you need to understand the relationship between variables and objects:

Variable: refers to an object through a variable pointer. The variable pointer points to the memory space of the specific object and takes the value of the object.

Object, the type is known, and each object contains a header information (header information: type identifier and reference counter)

Reference count

Everything in python is an object, and at its core is a structure: PyObject, where ob_refcnt is the reference count. When an object has a new reference, ob_refcnt increases, and when the object referencing it is deleted, the ob_refcnt decreases. When the reference count is 0, the life of the object ends.

Typedef struct_object {int ob_refcnt; struct_typeobject * ob_type;} PyObject;#define Py_INCREF (op) ((op)-> ob_refcnt++) / / increase the count # define Py_DECREF (op)\ / / decrease the count if (--(op)-> ob_refcnt! = 0)\;\ else\ _ Py_Dealloc ((PyObject *) (op))

You can use the sys.getrefcount () function to get the reference count of an object, but it should be noted that it will be used 1 more times than expected, because a temporary reference is automatically generated for the queried object when called.

Here is a brief look at the process of changing the reference count.

Start by creating three objects with a reference count of 1.

After pointing N1 to the new object "JKL", the reference count of the previous object "ABC" becomes 0. At this point, Python's garbage collector starts working, releasing the "ABC".

Next, let N2 refer to N1. "DEF" is no longer referenced, and "JKL" becomes 2 because it is referenced by N1 and N2 at the same time.

> > N1 = "ABC" > N2 = "DEF" > N2 = "GHI" > sys.getrefcount (N1) 2 > > N1 = "JKL" > sys.getrefcount (N1) 2 > > N2 = N1 > sys.getrefcount (N1) 3 > sys.getrefcount (N2) 3 > sys.getrefcount (N2) 2:

Advantages: good real-time performance. Once there is no reference, the memory is directly freed. Real-time also brings another benefit: the time spent on processing reclaimed memory is apportioned to normal times.

Disadvantages: maintaining reference counts consumes resources; circular references cannot be resolved.

As shown in the following figure, a typical circular reference scenario. In addition to being referenced by variables N1 and N2, objects are also referenced by each other's prev or next pointers, resulting in a reference count of 2. After N1 and N2 are set to null, the reference count is still 1, so that the object cannot be recycled.

Mark-clear, generational collection

Python uses a mark-and-clear strategy to solve the problem of circular references. However, this mechanism will cause the application to get stuck, and in order to reduce the pausing time of the program, the garbage collection efficiency is improved by the method of exchanging space for time through "Generational Collection". See Python garbage collection mechanism for details! Very practical

Reference count of Python C extension

Python provides a GC mechanism to ensure that objects will be released when they are not used, and developers do not need to care too much about memory management. But when using C # extensions, it's not that simple, and you have to understand the reference count of CPython.

When using Python with the C extension, the reference count automatically increases by 1 with the creation of the PyObjects, but when we release the PyObjects, we need to display to subtract the reference count of the PyObjects by 1, otherwise there will be a memory leak.

# include "Python.h" void print_hello_world (void) {PyObject * pObj = NULL; pObj = PyBytes_FromString ("Hello world\ n"); / * Object creation, ref count = 1. * / PyObject_Print (pLast, stdout, 0); Py_DECREF (pObj); / * ref count becomes 0, object deallocated. * Miss this step and you have a memory leak. * /}

There are some bright spots that need to be noted in particular:

After the PyObjects reference count is 0, it can no longer be accessed. Similar to the C language free, objects can no longer be accessed.

Py_INCREF and Py_DECREF must appear in pairs. Similar to the relationship between C language malloc and free.

There are three forms of Python references, namely "New", "Stolen" and "Borrowed" references.

New reference

With a PyObject created by Python C Api, the caller has full ownership of the PyObject. General Python documents are shown as follows:

PyObject* PyList_New (int len) Return value: New reference. Returns a new list of length len on success, or NULL on failure.

There are two options for PyObject referenced by New. Otherwise, there will be a memory leak.

After the use is complete, call Py_DECREF to release it.

Void MyCode (arguments) {PyObject * pyo;... Pyo = Py_Something (args);... Py_DECREF (pyo);}

The reference is passed to the upper calling function in the form of a function return value, etc., but the receiver must be responsible for the final Py_DECREF call.

Void MyCode (arguments) {PyObject * pyo;... Pyo = Py_Something (args);... return pyo;}

Use the sample:

Static PyObject * subtract_long (long a, long b) {PyObject * pA, * pB, * r; pA = PyLong_FromLong (a); / * pA: New reference. * / pB = PyLong_FromLong (b); / * pB: New reference. * / r = PyNumber_Subtract (pA, pB); / * r: New reference. * / Py_DECREF (pA); / * My responsibility to decref. * / Py_DECREF (pB); / * My responsibility to decref. * / return r; / * Callers responsibility to decref. * /} / / error example, a, b two PyObject leaks. R = PyNumber_Subtract (PyLong_FromLong (a), PyLong_FromLong (b)); Stolen reference

When the created PyObject is passed to other containers, such as PyTuple_SetItem, PyList_SetItem.

Static PyObject * make_tuple (void) {PyObject * r; PyObject * v; r = PyTuple_New (3); / * New reference. * / v = PyLong_FromLong (1L); / * New reference. * / * PyTuple_SetItem "steals" > PyTuple_SetItem (r, 0, v); / * This is fine. * / v = PyLong_FromLong (2L); PyTuple_SetItem (r, 1, v); / * More common pattern. * / PyTuple_SetItem (r, 2, PyUnicode_FromString ("three")); return r; / * Callers responsibility to decref. * /}

However, it is important to note that the reference count is incremented by one within PyDict_SetItem.

Borrowed reference

In Python documents, Borrowed references are reflected as follows:

PyObject* PyTuple_GetItem (PyObject* p, Py_ssize_t pos) Return value: Borrowed reference.

The owner of the Borrowed reference should not call Py_DECREF (), and using the Borrowed reference will not cause a memory leak when the function exits. But don't let an object handle an unprotected state Borrowed reference. If an object handles an unprotected state, it may be destroyed at any time.

For example: get an object from a list and continue to manipulate it without incrementing its reference. PyList_GetItem returns a borrowed reference, so the item is unprotected. Some other operation may remove the object from the list (decrement its reference count or release it), causing the item to become a dangling pointer.

Bug (PyObject * list) {PyObject * item = PyList_GetItem (list, 0); PyList_SetItem (list, 1, PyInt_FromLong (0L)); PyObject_Print (item, stdout, 0); / * BUG! * /} no_bug (PyObject * list) {PyObject * item = PyList_GetItem (list, 0); Py_INCREF (item); / * Protect item. * / PyList_SetItem (list, 1, PyInt_FromLong (0L)); PyObject_Print (item, stdout, 0); Py_DECREF (item);} Thank you for reading, the above is the content of "the analysis of the citation counting problem of Python C extension". After the study of this article, I believe you have a deeper understanding of the analysis of the citation counting problem of Python C extension, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.