Shulou(Shulou.com)06/02 Report--

This article mainly explains the "Python multi-process multiprocessing package what is the role of", the article explains the content is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in-depth, together to study and learn "Python multi-process multiprocessing package what is the role of it?"

Threading and multiprocessing

The multiprocessing package is a multi-process management package in Python. Similar to threading.Thread, it can use the multiprocessing.Process object to create a process. The process can run functions written within the Python program. This Process object is used in the same way as the Thread object, and there are also methods for start (), run (), join (). In addition, there are Lock/Event/Semaphore/Condition classes in the multiprocessing package (these objects can be passed to individual processes through parameters like multithreading) to synchronize processes, using the same as the classes of the same name in the threading package. As a result, a large part of multiprocessing uses the same set of API as threading, but in a multi-process context.

However, when using these shared API, we should pay attention to the following points:

On the UNIX platform, when a process terminates, the process needs to be called wait by its parent process, otherwise the process becomes a Zombie. Therefore, it is necessary to call the join () method (effectively equivalent to wait) on each Process object. For multithreading, this is not necessary because there is only one process.

Multiprocessing provides IPC (such as Pipe and Queue) that are not available in the threading package, which is more efficient. Pipe and Queue should be given priority, and synchronization methods such as Lock/Event/Semaphore/Condition should be avoided (because they do not occupy the resources of the user process).

Multiple processes should avoid sharing resources. In multithreading, we can easily share resources, such as using global variables or passing parameters. In the case of multiple processes, the above approach is not appropriate because each process has its own independent memory space. At this point, we can share resources by sharing memory and Manager. However, this increases the complexity of the program and reduces the efficiency of the program because of the need for synchronization.

PID is stored in Process.PID, and PID is None if the process does not already have start ().

We can see the similarities and differences between Thread objects and Process objects in usage and results in the following program. Each thread and process does one thing: print the PID. The problem is that all tasks are printed to the same standard output (stdout). In this way, the output characters will be mixed and cannot be read. Using Lock synchronization, after one task output is completed, another task output is allowed, which can avoid multiple tasks being output to the terminal at the same time.

# Similarity and difference of multi thread vs. Multiprocess # Written by Vameiimport osimport threadingimport multiprocessing# worker functiondef worker (sign, lock): lock.acquire () print (sign, os.getpid ()) lock.release () # Mainprint ('Main:',os.getpid ()) # Multi-threadrecord = [] lock = threading.Lock () for i in range (5): thread = threading.Thread (target=worker,args= (' thread') Lock)) thread.start () record.append (thread) for thread in record: thread.join () # Multi-processrecord = [] lock = multiprocessing.Lock () for i in range (5): process = multiprocessing.Process (target=worker,args= ('process',lock)) process.start () record.append (process) for processin record: process.join ()

The PID of all Thread is the same as the main program, and each Process has a different PID.

(exercise: use the mutiprocessing package to change Python multithreaded and synchronized multithreaded programs to multiprocess programs)

Pipe and Queue

As we introduced in Linux multithreading, the pipe PIPE and message queuing message queue,multiprocessing packages have Pipe classes and Queue classes to support these two IPC mechanisms, respectively. Pipe and Queue can be used to transfer common objects.

1) Pipe can be unidirectional (half-duplex) or bi-directional (duplex). We create an one-way pipe through mutiprocessing.Pipe (duplex=False) (bidirectional by default). A process inputs objects from one end of the PIPE and is received by processes on the other side of the PIPE. One-way pipes only allow processes on one side of the pipe to input, while two-way pipes allow input from both ends.

The following program shows the use of Pipe:

# Multiprocessing with Pipe# Written by Vameiimport multiprocessing as muldef proc1 (pipe): pipe.send ('hello') print (' proc1 rec:',pipe.recv ()) def proc2 (pipe): print ('proc2 rec:',pipe.recv ()) pipe.send (' hello, too') # Build a pipepipe = mul.Pipe () # Pass an end of the pipe to process 1p1 = mul.Process (target=proc1, args= (pipe [0],)) # Pass the other end of the pipe to process 2p2 = mul.Process (target=proc2 Args= (pipe [1],) p1.start () p2.start () p1.join () p2.join ()

The Pipe here is two-way.

When the Pipe object is created, it returns a table with two elements, each representing one end of the Pipe (the Connection object). We call the send () method on one end of the Pipe to pass the object and use recv () to receive it on the other side.

2) Queue is similar to Pipe in that it is a first-in-first-out structure. However, Queue allows multiple processes to put in, and multiple processes take objects out of the queue. Queue is created using mutiprocessing.Queue (maxsize), and maxsize represents the maximum number of objects that can be stored in the queue.

The following program shows the use of Queue:

# Written by Vameiimport osimport multiprocessingimport time#=# input workerdef inputQ (queue): info = str (os.getpid ()) +'(put):'+ str (time.time ()) queue.put (info) # output workerdef outputQ (queue Lock): info = queue.get () lock.acquire () print (str (os.getpid ()) +'(get):'+ info) lock.release () # = # Mainrecord1 = [] # store input processesrecord2 = [] # store output processeslock = multiprocessing.Lock () # To prevent messy printqueue = multiprocessing.Queue (3) # input processesfor i in range (10): process = multiprocessing.Process (target=inputQ,args= (queue) ) process.start () record1.append (process) # output processesfor i in range (10): process = multiprocessing.Process (target=outputQ,args= (queue,lock)) process.start () record2.append (process) for p in record1: p.join () queue.close () # No more object will come, close the queuefor p in record2: p.join ()

Some processes use put () to put a string in Queue that contains PID and time. Other processes take it out of the Queue and print their own PID and the string of get ().

Thank you for your reading, the above is the "Python multi-process multiprocessing package what is the role of" the content, after the study of this article, I believe you have a deeper understanding of the role of Python multi-process multiprocessing package, the specific use of the situation also needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.