Shulou(Shulou.com)06/02 Report--

This article mainly explains "Process class analysis of multiprocessing module in Python". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "Process class analysis of multiprocessing module in Python".

A new multiprocessing module has been added to the Python2.6 version. It was originally defined in PEP 371by Jesse Noller and Richard Oudkerk. Just as you can derive threads through the threading module, the multiprocessing module allows you to derive processes. The idea used here: because you can now derive processes, you can avoid using global interpreter locks (GIL) and make full use of the machine's multiple processors.

The multiprocess package also contains some API that is not even in the threading module. For example, there is a flexible Pool class that allows you to execute functions in parallel under multiple inputs. We will explain the Pool class in a later section. We'll start with the Process class of the multiprocessing module.

Start learning the multiprocessing module

The Process class is similar to the Thread class in the threading module. Let's create a series of processes that call the same function and see how it works.

Import os from multiprocessing import Process def doubler (number): "" A doubling function that can be used by a process "" result = number * 2 proc = os.getpid () print ('{0} doubled to {1} by process id: {2} '.format (number, result, proc)) if _ _ name__ = =' _ _ main__': numbers = [5, 10, 15, 20 25] procs = [] for index, number in enumerate (numbers): proc = Process (target=doubler, args= (number,)) procs.append (proc) proc.start () for proc in procs: proc.join ()

For the above example, we import the Process class and create a function called doubler. In the function, we multiply the passed number by 2. We also use Python's os module to get the ID (pid) of the current process. This ID will tell us which process is calling the doubler function. Then, in the following code block, we instantiate a series of Process classes and start them. * A loop simply calls the join () method of each process, which tells Python to wait for the process until it ends. If you need to end a process, you can call its terminate () method.

When you run the above code, you should see an output similar to the following:

5 doubled to 10 by process id: 10468 10 doubled to 20 by process id: 10469 15 doubled to 30 by process id: 10470 20 doubled to 40 by process id: 10471 25 doubled to 50 by process id: 10472

Sometimes you give your process an easy-to-understand name. Fortunately, the Process class does allow you to access the same process. Let's take a look at the following examples:

Import os from multiprocessing import Process, current_process def doubler (number): "" A doubling function that can be used by a process "result = number * 2 proc_name = current_process (). Name print ('{0} doubled to {1} by: {2} '.format (number, result, proc_name)) if _ name__ = =' _ main__': numbers = [5, 10, 15, 20 25] procs = [] proc = Process (target=doubler, args= (5,)) for index, number in enumerate (numbers): proc = Process (target=doubler, args= (number,) procs.append (proc) proc.start () proc = Process (target=doubler, name='Test', args= (2,) proc.start () procs.append (proc) for proc in procs: proc.join ()

This time, we imported more current_process. Current_process is basically similar to the current_thread of the threading module. We use it to get the name of the thread that is calling our function. You will notice that we did not set names for the previous five processes. Then we set the name of the sixth process to "Test".

Let's see what kind of output we will get:

5 doubled to 10 by: Process-2 10 doubled to 20 by: Process-3 15 doubled to 30 by: Process-4 20 doubled to 40 by: Process-5 25 doubled to 50 by: Process-6 2 doubled to 4 by: Test

The output shows that by default, the multiprocessing module assigns a number to each process, and that number is used to form part of the process's name. Of course, if we give a name, no number will be added to the name.

Lock

The multiprocessing module supports locks in the same way that the threading module does. All you have to do is import the Lock, get it, do something, and release it.

From multiprocessing import Process, Lock def printer (item, lock): "Prints out the item that was passed in" lock.acquire () try: print (item) finally: lock.release () if _ _ name__ ='_ _ main__': lock = Lock () items = ['tango',' foxtrot' " 10] for item in items: P = Process (target=printer, args= (item, lock)) p.start ()

Here we have created a simple print function that outputs whatever you enter. To prevent threads from blocking each other, we use the Lock object. Loop through the three items in the list and create a process for each of them. Each process will call our function, and the item traversed each time will be passed into the function as an argument. Because we now use locks, the next process in the queue will block until the previous process releases the lock.

Journal

There are some differences between creating logs for processes and creating logs for threads. They are different because Python's logging packages do not use processes with shared locks, so it is possible to end with information from different processes. Let's try to add a basic log to the previous example. The code is as follows:

Import logging import multiprocessing from multiprocessing import Process, Lock def printer (item, lock): "Prints out the item that was passed in" lock.acquire () try: print (item) finally: lock.release () if _ _ name__ ='_ _ main__': lock = Lock () items = ['tango',' foxtrot' " 10] multiprocessing.log_to_stderr () logger = multiprocessing.get_logger () logger.setLevel (logging.INFO) for item in items: P = Process (target=printer, args= (item, lock)) p.start ()

The easiest way to add a log is by pushing it to stderr. We can implement this method by calling the thelog_to_stderr () function. Then we call the get_logger function to get an instance of logger and set its log level to INFO. The code after that is the same. I need to be reminded that I didn't call the join () method here. Instead: when it exits, the parent thread automatically calls the join () method.

When you do this, you should get output similar to the following:

[INFO/Process-1] child process calling self.run () tango [INFO/Process-1] process shutting down [INFO/Process-1] process exiting with exitcode 0 [INFO/Process-2] child process calling self.run () [INFO/MainProcess] process shutting down foxtrot [INFO/Process-2] process shutting down [INFO/Process-3] child process calling self.run () [INFO/Process-2] process exiting with exitcode 0 10 [INFO/MainProcess] calling join () for process Process-3 [INFO/Process-3] process shutting down [INFO/Process-3] process exiting with exitcode 0 [INFO/MainProcess] calling join () for process Process-2

Now if you want to save the log to your hard drive, it's a bit tricky. You can read about that kind of topic at Python's logging Cookbook.

Pool class

The Pool class is used to represent a worker process pool. It has a way for you to transfer tasks to the work process. Let's look at the following very simple example.

From multiprocessing import Pool def doubler (number): return number * 2 if _ _ name__ = ='_ main__': numbers = [5,10,20] pool = Pool (processes=3) print (pool.map (doubler, numbers))

Basically, after the above code is executed, an instance of Pool is created, and the instance creates three worker processes. Then we use the map method to map a function and an iterable object to each process. * We print out the result of this example: [10, 20, 40].

You can also get the running results of the processes in the pool through the apply_async method:

From multiprocessing import Pool def doubler (number): return number * 2 if _ _ name__ = ='_ _ main__': pool = Pool (processes=3) result = pool.apply_async (doubler, (25,)) print (result.get (timeout=1))

What we do above is actually the running result of the request process. That's what the get function is for. It tries to get our results. You can notice that we set timeout to prevent the exception of the function we called. After all, we don't want it to be blocked.

Process communication

When it comes to interprocess communication, the multiprocessing module provides two main methods: Queues and Pipes. Queue implementation is both thread-safe and process-safe. Let's look at a fairly simple and Queue-based example. The code comes from my article threading articles.

From multiprocessing import Process, Queue sentinel =-1 def creator (data, Q): "Creates data to be consumed and waits for the consumer to finish processing" print ('Creating data and putting it on the queue') for item in data: q.put (item) def my_consumer (Q): "Consumes some data and works on it In this case" All it does is double the input "" while True: data = q.get () print ('data found to be processed: {}' .format (data)) processed = data * 2 print (processed) if data is sentinel: break if _ _ name__ = ='_ main__': Q = Queue () data = [5,10,13 -1] process_one = Process (target=creator, args= (data, Q)) process_two = Process (target=my_consumer, args= (Q,)) process_one.start () process_two.start () q.close () q.join_thread () process_one.join () process_two.join ()

Here we just need to import Queue and Process. Queue is used to create data and add data to queues, and Process is used to consume data and execute it. By using the put () and get () methods of Queue, we can add data to Queue and get data from Queue. The * * block of the code simply creates the Queue object and two Process objects and runs them. You'll notice that we call the join () method on the process object instead of calling it on the Queue itself.

Thank you for your reading, the above is the content of "Process class analysis of multiprocessing module in Python". After the study of this article, I believe you have a deeper understanding of the Process class analysis of multiprocessing module in Python, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.