Shulou(Shulou.com)06/01 Report--

This article mainly introduces the relevant knowledge of "what is the mode of communication between Python processes". The editor shows you the operation process through an actual case. The operation method is simple, fast and practical. I hope this article "what is the mode of communication between Python processes" can help you solve the problem.

What is process communication?

Here is an example to introduce the mechanism of communication: the word communication is not unfamiliar to everyone, for example, a person wants to call his girlfriend. When a call is established, an invisible queue is established during the call (remember the word). At this point, the person will keep telling his girlfriend the information through dialogue, and the person's girlfriend is listening. (um... Personally, I think it may be the opposite in most cases.

Here, the two of them can be compared to two processes. The process of "this person" needs to send information to the process of "girlfriend" and needs the help of a queue. On the other hand, the girlfriend needs to constantly receive the information from the queue and can do some other things, so the communication between the two processes mainly depends on the queue.

This queue can support sending and receiving messages. "this person" is responsible for sending messages, while "girlfriend" is responsible for receiving messages.

Since queues are the key, let's take a look at how queues are created.

Queue creation-multiprocessing

The multiprocessing module is still used, and the Queue function of the module is called to create the queue.

Function name describes the parameter return value Queue queue creation mac_count queue object

Queue function description: call Queue to create a queue; it has a parameter mac_count that represents the maximum number of information that can be created for the queue. If it is not passed, the default length is unlimited. After instantiating a queue object, you need to manipulate the queue object to put and retrieve data.

The function name of the method for communication between processes introduces the parameter return value put puts messages into queues message no get get queue messages no str

Introduction to the function of put function: pass in data. It has a parameter message, which is a string type.

Get function introduction: used to receive data in the queue. In fact, this is a commonly used json scenario, there are a lot of data transmission is a string, queue insertion and acquisition is the use of strings, so json is very suitable for this scenario. )

Next let's practice the use of queues.

Inter-process communication-queue demonstration case

The code example is as follows:

# coding:utf-8import jsonimport multiprocessingclass Work (object): # define a Work class def _ init__ (self, queue): # pass a 'queue object'-> queue self.queue = queue def send (self, message): # define a send (send) function Pass in message # [there is a hidden bug that only determines whether the passed in is of string type If not isinstance (message, str): # determines whether the passed message is a string, if not Then json serialize message = json.dumps (message) self.queue.put (message) # send the message out to def receive (self) using the queue instantiation object of queue: # define a receive (receive) function without passing in parameters, but because receiving is a continuous process So we need to use the while loop while 1: result = self.queue.get () # to get the 'queue object'-> the message # passed in by queue, because the message we receive may not be a string So process exception capture try: # if the message passed in conforms to JSON format, it will be assigned to res If it doesn't match, Then directly use result to assign res res = json.loads (result) except: res = result print ('the message received is: {}' .format (res)) if _ _ name__ = ='_ main__': queue = multiprocessing.Queue () work = Work (queue) send = multiprocessing.Process (target=work.send) Args= ({'message':' this is a test message'},) receive = multiprocessing.Process (target=work.receive) send.start () receive.start ()

Exceptions encountered in establishing interprocess communication using queues

But there will be an error, as shown in the following figure:

An example of screenshot of an error is as follows:

The error message here means that the file has not been found. In fact, when we use queues to do put () and get (), we have an invisible lock on it, which is the .SemLock in the circle in the image above. We do not need to pay attention to the specific cause of this error, it is actually very simple to solve this problem.

FileNotFoundError: [Errno 2] solution to No such file or directory exception

We only need to add a join blocking process to one of the child processes of send or receive, in theory. But our receive child process is a while loop, and it executes all the time, so you only need to add a join to the send child process.

The schematic diagram of the solution is as follows:

PS: although the error problem was solved, the program did not exit normally.

In fact, because our receive process is a while loop, we don't know when to deal with it, so there is no way to stop it immediately. So we need to use the terminate () function to terminate the receiver in the receive process.

The running results are as follows:

Add data to send function in batch

Create a new function and write it into the for loop to simulate batch adding messages to be sent

Then add a thread to the function that simulates sending data in bulk.

The sample code is as follows:

# coding:utf-8import jsonimport timeimport multiprocessingclass Work (object): # define a Work class def _ init__ (self, queue): # pass a 'queue object'-> queue self.queue = queue def send (self, message): # define a send (send) function Pass in message # [there is a hidden bug that only determines whether the passed in is of string type If not isinstance (message, str): # determines whether the passed message is a string, if not Then json serialize message = json.dumps (message) self.queue.put (message) # send the message out to def send_all (self) using the queue instantiation object of queue: # define a send_all (send) function Then the batch message for i in range (20): self.queue.put ('{} cycle') is simulated through the for loop, and the message sent is: {} '.format (I, I) time.sleep (1) def receive (self): # defines a receive (receive) function without passing in parameters, but because receiving is a continuous process So we need to use the while loop while 1: result = self.queue.get () # to get the 'queue object'-> the message # passed in by queue, because the message we receive may not be a string So process exception capture try: # if the message passed in conforms to JSON format, it will be assigned to res If it doesn't match, Then directly use result to assign res res = json.loads (result) except: res = result print ('the message received is: {}' .format (res)) if _ _ name__ = ='_ main__': queue = multiprocessing.Queue () work = Work (queue) send = multiprocessing.Process (target=work.send) Args= ({'message':' this is a test message'},) receive = multiprocessing.Process (target=work.receive) send_all = multiprocessing.Process (target=work.send_all,) send_all.start () # here because send was executed only once And then it's over. While send_all loops 20 times, its execution time is the longest, and the send.start () receive.start () # send.join () # blocking using send will cause the receiver of the receive.terminate () function to terminate before the end of the send_all loop. Send_all.join () # so we only need to block the process with the longest usage and receive.terminate ()

The running results are as follows:

From the figure above, we can see that both send and send_all processes can send messages through the instantiated Queue object queue, and the same receive receiver function will print out the message passed in by the two processes.

Bar

In this chapter, we realize the method of inter-process communication by queuing, and understand how to use queues. In a queue, one end (here we demonstrate the send side) adds relevant information through the put method, and the other side uses the get method to obtain the relevant information; the two processes cooperate with each other to achieve the effect of a process communication.

In fact, the communication between processes is not only a way of queue, but also through pipes, semaphores and shared memory if you are interested. You can expand on your own.

Other ways of interprocess communication-complementary

Python provides a variety of ways for processes to communicate, including signals, pipes, message queues, semaphores, shared memory, socket, etc.

There are two main ways: Queue and Pipe, Queue is used for communication between multiple processes, and Pipe is the communication between two processes.

1. Pipes: divided into anonymous pipes and named pipes

Anonymous pipeline: apply for a fixed size buffer in the kernel. The program has the right to write and read. Generally, the fock function is used to realize the communication between the parent and child processes.

Named pipe: apply for a fixed size buffer in memory, the program has the right to write and read, and unrelated processes can communicate between processes.

Features: oriented to byte flow; life cycle with kernel; built-in synchronous mutex mechanism; half-duplex, one-way communication, two-way communication between two pipes

two。 Message queue: create a queue in the kernel, each element in the queue is a Datagram, and different processes can access the queue through handles. Message queuing provides a way to send a piece of data from one process to another. Each data block is considered to have a type, and the data blocks received by the receiver process can have different types. Message queues also have the same deficiency as pipes, that is, the maximum length of each message has an upper limit, the total number of bytes of each message queue has an upper limit, and the total number of message queues on the system also has an upper limit.

Features: message queue can be regarded as a global linked list, the type and content of Datagram are stored in the linked list node, marked by the identifier of message queue; message queue allows one or more processes to write or read messages; the life cycle of message queue follows the kernel; message queuing enables two-way communication

3. Semaphores: create a set of semaphores in the kernel (essentially an array). The elements of the array (semaphores) are all 1, using P operation for-1, and V operation + 1.

P (sv): if the value of sv is greater than zero, subtract it by 1; if its value is zero, suspend the execution of the program

V (sv): if another process is suspended while waiting for sv, let it resume running. If no process is suspended while waiting for sv, add 1 to it.

PV operation is used for the same process to achieve mutual exclusion, while PV operation is used for different processes to achieve synchronization

Function: protect critical resources

4. Shared memory: map the same piece of physical memory to the virtual address space of different processes to realize the sharing of the same resource among different processes. Shared memory is arguably the most useful way of interprocess communication, and it is also the fastest IPC form.

Features: different frequent switching and copying data from user mode to kernel state can be read directly from memory; shared memory is a critical resource, so atomicity must be guaranteed when operation is needed. You can use semaphores or mutexes.

This is the end of the content about "what is the mode of communication between Python processes". Thank you for your reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.