Shulou(Shulou.com)06/02 Report--

This article mainly introduces the relevant knowledge of Python garbage collection and Linux Fork case analysis, the content is detailed and easy to understand, the operation is simple and fast, and has a certain reference value, I believe you will have something to gain after reading this Python garbage collection and Linux Fork case analysis article, let's take a look.

Foreword:

The pocket assistant saw a little bit of optimization work done by colleagues in other departments for Python2 memory footprint, and he was more interested, so he recorded it.

Introduction to 1.Linux fork

Fork is a system call to create a child process provided by Linux. In order to optimize the speed of the creation process, the Linux kernel uses Copy-on-Write to create the process. The so-called Copy-on-Write refers to the execution of the fork.

Instead of immediately allocating physical memory space to the child process, the kernel maps the child process's virtual memory to the parent process's physical memory. A section of physical memory is allocated to a child process only when it is writing to the address space.

In this way, it not only optimizes the process creation time, but also reduces the memory footprint of child processes.

The reason why 1.Copy-On-Write policy increases the memory consumption of Python multi-processes

Python GC uses reference technology to manage references to each object, and each object tracked by GC is represented by a PyGC_Head structure. As shown below, where gc_refs is the reference count value for each object

When we read the object created by the parent process in the child process, it will cause the gc_refs in the virtual address space of the child process to add 1, thus triggering the missing page interrupt of the kernel, which means that the kernel will create new physical memory for the child process.

A simple read operation can lead to new memory space.

/ * GC information is stored BEFORE the object structure. * / typedef union _ gc_head {struct {union _ gc_head * gc_next; union _ gc_head * gc_prev; Py_ssize_t gc_refs;} gc; long double dummy; / * force worst-case alignment * /} PyGC_Head;3. Solution.

The solution to python3:

To solve this problem, Python3.7 added three groups of API (submitted by the instagram group) [1].

Freeze is used to move all objects tracked by GC to permanent generation, after which garbage collection ignores objects that are set to immortal.

In practice, we can execute the freeze function in the parent process, and then use the object shared with the parent process in the child process, so that the object reference technology will not be increased, thus avoiding the occurrence of COW.

The solution to python2:

(1) for Python2, we can simply transplant the related functions of Python3.

(2) use multiprocessing.Array to share data. Array fetches data from shared memory and does not increase the reference technology value, thus triggering COW.

In terms of implementation, Array uses Posix shared memory + mmap to implement. [3]

#! / usr/bin/env python# coding=utf-8from multiprocessing import Arrayimport osimport sysdef foo (): shared_cache = Array ('parent, range (0100), lock=False) pid = os.fork () if pid > 0: print ("parent:", sys.getrefcount (shared_cache)) elif pid = 0: print ("child:" Sys.getrefcount (shared_cache)) foo () introduces the article "Python garbage collection and Linux Fork instance analysis" here. Thank you for reading! I believe you all have some knowledge about "Python garbage collection and Linux Fork case analysis". If you want to learn more, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.