Shulou(Shulou.com)06/01 Report--

This article mainly introduces the relevant knowledge of "what is the basic concept of Python process pool". The editor shows you the operation process through an actual case. The operation method is simple, fast and practical. I hope this article "what is the basic concept of Python process pool" can help you solve the problem.

Foreword:

Creating a process pool can be vividly understood as creating a parallel pipeline, which only needs to create a consumption of the pipeline once, and does not use the process pool to handle received tasks. Oh, waste of time.

There is no process on the Chinese side, except for python, which uses the thread pool language, which is the other thread pool of the process (and the process is the other task of executing the business). Because of the concept of python (because of the concept of Cython), thread programming in different parallelism shifts the concept of thread pool to the process, which is named process pool.

1. Python process pool

When the number of child processes created is small, you can directly use the processes in the multiprocessing process to dynamically form the desired processes.

If it is hundreds or even thousands, manually create the working goal of the process, at this time you can provide a pool method for the multi-process module.

1. When initializing Pool, you can specify a number of processes

2. When a new request is submitted to the Pool

If the pool is not full, the request is executed by creating a new process

If the process in the pool reaches the specified task, how much time has been left until the process ends, the previous requesting process will be used to perform the new task.

Second, how to use process pool? Application ()

Function prototype: apply (func, args= () [, kwds= {}]])

This function passes variable parameters, which is consistent with the application function in python, and the main process will be blocked from executing the function (not recommended, and will not appear after 3.x).

Apply_async

Function prototype: apply_async (func [, args= () [, kwds= {} [, callback=None])

Consistent with the application, but it is not supported by the use of support after the result is returned.

Map ()

Function prototype: map (func, iterable [, chunksize=None])

The behavior of the mapping table in the pool class must be basically the same, and it uses the second process to return with the previous result:

But in practice, a parameter is a call, and it needs to be noted throughout the application that the program runs the child process.

Map_async ()

Function prototype: map_async (func, iterable [, chunksize [, callback]])

It matches the map, but it is undamped.

Close ()

Close the process pool (pool) and no longer accept new tasks.

Terminal ()

End the work process and no longer deal with unhandled tasks.

Add ()

The main process stops and waits for the exit of the child process, and the joining method should be closed or terminated.

3. The code is listed # Import the relevant multiprocessing package import multiprocessing# to create a process pool with the number of processes with CPU cores pool = multiprocessing.Pool (processes=multiprocessing.cpu_count ()) for i in range: # # stop the process waiting for the current task to end # pool.apply (func=pow, args=) # do not delay the process end of the current task pool.apply_async (func=pow, args= (I) 2) # # map function to a list The delayed return value # results = pool.map (func=print, iterable= [i for i in range (10000)]) # # does not delay waiting for the return value. Calling results before running will report an error. # results = poolmap_async (func=print, iterable= [i for i in range (10000)]) # close No new processes will be added to pool pool.close () # join the function and wait for all child processes # stop calling the close function before joining, otherwise an error will occur. Pool.join () # # ends the work process and no longer handles unfinished tasks. # pool.terminate () 4. What's the difference between a process in a process pool and a normal process?

Queue in the process pool:

If you want to create a process using a process pool, you need to use Queue () in multiprocessing.Manager () instead of multiprocessing.Queue ().

The method is the same, and it's all successful, then add it with q.put () and wait for it to get with q.get ().

This is the end of the introduction to "what is the basic concept of Python process pool". Thank you for reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.