Shulou(Shulou.com)06/01 Report--

This article focuses on "Python process pool and process lock example analysis", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Now let the editor to take you to learn "Python process pool and process lock instance analysis"!

Process pool what is a process pool

As we mentioned in the previous section on processes, too many processes can consume too much resources. To avoid this, we need a fixed number of processes, which requires the help of a process pool.

We can think of a process pool as a pool in which a certain number of processes are created in advance. See the following figure:

For example, this red rectangular array represents a process pool in which there are six processes. These six processes are created along with the process pool, and not only that, we said when learning about the object-oriented life cycle that each instantiated object is reclaimed by the memory butler after it has been used.

Our processes are also recycled by the memory butler along with the process of creating and closing, each of which is the case, and the process of creating a closed process consumes some performance. On the other hand, the processes in the process pool will not be closed after they are created and can be reused all the time, thus avoiding the resource consumption created in the closure and improving the efficiency by avoiding the repeated operations created in the closure.

Of course, when we finish shutting down the process pool, the process shuts down.

When we have tasks that need to be executed, we determine whether there are any idle processes in the current process pool (the so-called idle processes are actually processes that do not execute tasks in the process pool). When a process is idle, the task will find the process to execute the task. If all the processes in the current process pool are not idle, the task will enter the waiting state and will not enter or leave the process pool until there is a process idle in the process pool to execute the task.

This is what process pools do.

Process pool creation module-multiprocessing create process pool function-Pool function name introduces the parameter return value Pool process pool creation Processcount process pool object

Pool function description: by calling the "multiprocessing" module of the "Pool" function to help us create a "process pool object", it has a parameter "Processcount" (an integer), on behalf of our process pool to create several processes.

Common methods of process pool

After the process pool object is created, we will process it. Let's take a look at what common methods (functions) are available.

Function name introduction parameter return value apply_async task join process pool (asynchronous) func,args no close close process pool no join wait for process pool task to end nothing

Apply_async function: its function is to add tasks to the process pool and is implemented asynchronously. We haven't learned the knowledge of asynchronism yet, so we don't have to worry about what it means. It has two parameters: func and agrs, func is a function that joins the process pool, and args is a tuple that represents the parameters of a function, which is exactly the same as when we create and use a process.

Close function: after we have finished using the process pool, we can close the process pool by calling the close function. It does not have any parameters and does not have any return value.

The join function: this is consistent with the method in the join function of the creation process we learned in the previous chapter. Subsequent tasks will not be executed until all the tasks in the process pool have been executed. However, it is usually used with the shutdown of the process pool (the close function).

Demonstration case of apply_async function

Next, let's create a script in Pycharm to practice how to use the process pool.

Define a function and print out the number of times the function is executed each time and the process number of that number

Defines the number of process pools. The maximum number of processes executed at a time is set for the process pool.

The sample code is as follows:

# coding:utf-8import osimport timeimport multiprocessingdef work (count): # define a work function that prints out the number of times each execution and the process number print of that number (the {} execution of the'\ 'work\' function The process number is {} '.format (count, os.getpid ()) time.sleep (3) # print (' *') if _ _ name__ = ='_ main__': pool = multiprocessing.Pool (3) # defines the number of processes in the process pool Execute at most three processes at a time for i in range (21): pool.apply_async (func=work, args= (I,)) # the parameter passed in is a tuple, because we only have one I parameter, so we have to write args= (I,) time.sleep (15) # the sleep time here must be added, otherwise the main process will have finished running before our process pool is run The corresponding process pool is also closed.

The running results are as follows:

From the figure above, we can see that each time three processes are run at a time, and the process number of each process is different, but if you look closely, you will find that there is the same process number, which indicates that the process number of the process pool is being reused. This proves that as we described above, processes in the process pool will not be shut down and can be used repeatedly.

And we can also see that 3 processes are executed every 3 seconds because there are only 3 processes in our process pool; although there are 21 tasks in our for loop and the work function is executed 21 times, because there are only 3 processes in our process pool. So after 3 tasks have been executed (hibernate for 3 seconds), the subsequent tasks wait for the processes in the process pool to be idle before they continue to execute.

Similarly, process numbers are somewhat different in order because we are using an asynchronous method (asynchronous, that is, asynchronous). This causes the three tasks executed by the work function to be out of order, which is why our process numbers appear in different order. (we will explain more about asynchrony in detail in the chapter on asynchrony.)

The principle of process pool: the case of the above script confirms the limitation of our process pool on processes, and only when the processes in our process pool are idle will the tasks waiting outside the process pool be thrown into the process pool to work.

Demonstration of close function and join function

In the script above, we used time.sleep (15) to help us block the main process for 15 seconds and exit again, so it gave us enough time for the process pool to complete the loop task of our work () function.

What if there is no time.sleep (15)? in fact, you can use the join function of the process here. However, as we mentioned above, the join () function of a process is usually used with the shutdown of the process pool (close function). Next, let's try replacing time.sleep (15) in the above script with the join () function.

The sample code is as follows:

# coding:utf-8import osimport timeimport multiprocessingdef work (count): # define a work function that prints out the number of times each execution and the process number print of that number (the {} execution of the'\ 'work\' function The process number is {} '.format (count, os.getpid ()) time.sleep (3) # print (' *') if _ _ name__ = ='_ main__': pool = multiprocessing.Pool (3) # defines the number of processes in the process pool Execute at most three processes at a time for i in range (21): pool.apply_async (func=work, args= (I,)) # the parameter passed in is a tuple, because we have only one I parameter, so we need to write as args= (I,) # time.sleep (15) pool.close () pool.join ()

The running results are as follows:

From the above dynamic diagram, we can see that the task of the work () function and the processes in the process pool are consistent with the results of running using time.sleep (15).

PS: if our main process will execute all the time, it won't quit. So we don't need to add the close () and join () functions to keep the process pool running until a task comes in and executes.

After learning WEB development later, it is common not to quit the main process to work. Some tasks that need to be executed for a long time will not be closed, but if there is only an one-time script, you need to add close () and join () functions to ensure that the main process exits after all the tasks of the process pool are completed. Of course, if the main process shuts down, it will not accept new tasks, which represents the end of the process pool.

Next, let's look at another example by adding a return to the work function.

Here you may have a question, in the previous chapter on the knowledge points of the process clearly said that the process can not get the return value, then here the work () function to add return what is the point?

In fact, when we use the apply_async method of the process pool, it is implemented asynchronously, which can get the return value. For the above script, we add a variable name for each asynchronous apply_async in the for loop to get the return value.

The sample code is as follows:

# coding:utf-8import osimport timeimport multiprocessingdef work (count): # define a work function that prints out the number of times each execution and the process number print of that number (the {} execution of the'\ 'work\' function The process number is {} '.format (count, os.getpid ()) time.sleep (3) return'\ 'work\' function result returns the value: {}, the process ID is: {} '.format (count, os.getpid ()) if _ _ name__ = =' _ main__': pool = multiprocessing.Pool (3) # defines the number of processes in the process pool Execute at most three processes at a time results = [] for i in range (21): result = pool.apply_async (func=work, args= (I,)) # the parameter passed in is a tuple, because we only have one I parameter So we need to write args= (I,) results.append (result) for result in results: print (result.get ()) # can return the return value of apply_async in this way, and # can execute normally without using the close () and join () functions. # time.sleep (15) # the sleep time here must be added, otherwise the main process will finish running before our process pool is running, and the corresponding process pool will be closed. # pool.close () # pool.join ()

The running results are as follows:

As can be seen from the running results, the work () function is first executed by the thread of the thread pool. When the first set of tasks is executed and then the second thread pool task is executed, the return value of apply_async is printed out, which proves that the return value has been returned successfully. And then move on to the next group of tasks.

These are mainly dependent on asynchrony, and more knowledge about asynchrony will be described in detail in the async section.

The concept of process lock

Lock: as we all know, we can lock a gate.

Let's take an example with this scenario: for example, there are multiple processes rushing toward a "gate" at the same time, when there is no "person" in the front door (in fact, the process), and the lock is not locked. When a process goes in and locks the door, those processes outside the door cannot get in. The "person" in the door can do anything in the "door" without being disturbed. When it comes out, it unlocks the door. At this time, another "person" enters the door and repeats this operation, which is the process lock. It allows the work behind the lock to be handled by only one task, and the next task will not enter until it is unlocked. This is the concept of "lock".

The process lock is only valid for the process, when the task of the process starts, it will be a "lock"; corresponding to the thread lock, their principle is almost the same.

Locking and unlocking of process lock

How to use process locks:

Import Manager classes through multiprocessing

From multiprocessing import Manager

Then instantiate the Manager

Manager = Manager ()

Then call its Lock () function through the instantiated manager

Lock = manager.Lock ()

Next, you need to manipulate the function of the lock object

Function name introduction parameter return value acquire lock no no release unlock (unlock) none

The code example is as follows:

# coding:utf-8import osimport timeimport multiprocessingdef work (count, lock): # define a work function, print out the number of times each execution and the process number of that number, and increase the thread lock. Lock.acquire () # Lock print (the {} execution of the'\ 'work\' function The process number is {} '.format (count, os.getpid ()) time.sleep (3) lock.release () # unlock return'\ 'work\' function result return value is: {}, process ID is: {} '.format (count, os.getpid ()) if _ _ name__ =' _ _ main__': pool = multiprocessing.Pool (3) # defines the number of processes in the process pool Execute at most three processes at a time manager = multiprocessing.Manager () lock = manager.Lock () results = [] for i in range (21): result = pool.apply_async (func=work, args= (I, lock)) # the parameter passed in is a tuple, because we only have one I parameter So we have to write args= (I,) # results.append (result) # time.sleep (15) # the sleep time here must be added, otherwise the main process will be finished before our process pool is running, and the corresponding process pool will be closed. Pool.close () pool.join ()

The implementation results are as follows:

From the figure above, you can see that only one task is performed at a time. Since each process is blocked for 3 seconds, our process executes very slowly. This is because each process enters the work () function and executes the process of locking, blocking for 3 seconds, and unlocking, thus completing the work of a process. The next process task begins, repeat the process. This is the concept of process locks.

In fact, there are many ways to lock a process. In multiprocessing, there is a lock that is directly used, which is ``from multiprocessing import Lock. The lock usage of this Lock is different from that of Manager`, which we just introduced. There is no detailed introduction here. If you are interested, you can expand it by yourself. )

The use of locks allows us to develop only one process for a task at a time, but locks can not be misused. Because if the lock is not unlocked properly for some reason, it will cause a deadlock, so it can no longer be operated.

Because if the lock can not be unlocked, there will be no way to continue to perform the task, so you must be careful when using the lock.

At this point, I believe you have a deeper understanding of "Python process pool and process lock instance analysis". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.