Shulou(Shulou.com)06/02 Report--

This article mainly introduces "what are the tools of Python multi-process multiprocessing package". In daily operation, I believe many people have doubts about what tools Python multi-process multiprocessing package has. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts of "Python multi-process multiprocessing package". Next, please follow the editor to study!

After a preliminary understanding of Python multiprocesses, we can continue to explore more advanced tools in the multiprocessing package. These tools make it easier for us to implement multiple processes.

Process pool

A process pool (Process Pool) can create multiple processes. These processes are like soldiers on standby, ready to carry out tasks. Multiple soldiers on standby can be held in a process pool.

"process pool for three processes"

For example, the following program:

Import multiprocessing as muldef f (x): return x**2pool = mul.Pool (5) rel = pool.map (f, [1 rel, 2, 3, 4, 5, 6, 7, 8, 9, 10])

We created a process pool (Process Pool) that allows five processes. Each process that Pool runs executes the f () function. We use the map () method to apply the f () function to each element of the table. This is similar to built-in 's map () function, except that it is processed in parallel with five processes. If there are still elements to be processed after the process is finished, then the process will be used to rerun the f () function. In addition to the map () method, Pool has the following common methods.

Apply_async (func,args) takes a process from the process pool to execute the parameter with func,args as func. It will return an object of AsyncResult on which you can call the get () method to get the result.

The close () process pool no longer creates new processes

Join () wait all processes in the process pool. You must call the close () method on Pool before you can join.

Practice

There is the following file, download.txt.

Www.sina.com.cnwww.163.comwww.iciba.comwww.cnblogs.comwww.qq.comwww.douban.com

Use a process pool containing 3 processes to download the home page of the Web site in the file. (you can use subprocess to call download tools such as wget or curl to perform specific download tasks.)

Share resources

As we have mentioned initially in Python, we should try to avoid sharing resources among multiple processes. The sharing of resources by multiple processes will inevitably lead to competition among processes. And this kind of competition will lead to race condition, and our results may be affected by the uncertainty of competition. But if necessary, we can still do this by sharing memory and Manager objects.

Shared memory

In Linux inter-process communication, we have described the principle of shared memory (shared memory). Here is an example of how to implement it in Python:

# modified from official documentationimport multiprocessingdef f (n, a): n.value = 3.14a [0] = 5num = multiprocessing.Value ('dumped, 0.0) arr = multiprocessing.Array (' iTunes, range (10)) p = multiprocessing.Process (target=f, args= (num, arr)) p.start () p.join () print num.valueprint arr [:]

Here we really only have the main process and the process represented by the Process object. We create shared memory in the memory space of the main process, that is, Value and Array objects. The object Value is set to a double number (d) and initialized to 0.0. Array is similar to an array in C, with a fixed type (I, that is, an integer). During the Process process, we modified the Value and Array objects. Back to the main program, the result is printed out, and the main program also sees a change in the two objects, indicating that the resources are indeed shared between the two processes.

Manager

The Manager object is similar to the communication (server-client) between the server and the client, and is very similar to our activity on Internet. We use a process as a server and set up a Manager to actually store resources. Other processes can access the Manager by passing parameters or by address, and then manipulate the resources on the server after establishing the connection. With the permission of the firewall, we can apply Manager to multiple computers, thus imitating a real network situation. In the following example, we use Manager similar to shared memory, but we can share richer object types.

Import multiprocessingdef f (x, arr, l): x.value = 3.14arr [0] = 5 l.append ('Hello') server = multiprocessing.Manager () x = server.Value (`dong.0.0) arr = server.Array (`iFountain, range (10)) l = server.list () proc = multiprocessing.Process (target=f, args= (x, arr, l) proc.start () proc.join () print (x.value) print (arr) print (l)

Manager provides a way to share tables using the list () method. You can actually use dict () to share dictionaries and Lock () to share threading.Lock (note that we share threading.Lock, not the mutiprocessing.Lock of the process. The latter itself has implemented process sharing) and so on. In this way, Manager allows us to share more diverse objects.

At this point, on the "Python multi-process multiprocessing package what tools" study is over, I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.