Shulou(Shulou.com)05/31 Report--

This article mainly explains the "Python multi-process knowledge points", the article explains the content is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "what are the Python multi-process knowledge points" bar!

First, what is multi-process?

1. Process

Program: for example, xxx.py, this is a program, it is a static

Process: after a program is running, the resource used by the code + is called the process, which is the basic unit for the operating system to allocate resources. Not only can multitasking be accomplished through threads, but processes can also be done.

two。 Status of the process

In work, the number of tasks is often greater than the core number of cpu, that is, there must be some tasks being executed, while others are waiting for cpu to be executed, which leads to different states.

Ready: the running conditions have slowed down and are waiting to be executed in cpu

Executive state: cpu is performing its function

Waiting state: waiting for certain conditions, such as a program sleep, to be in the waiting state.

Second, the creation of process-multiprocessing1. Process class syntax description

The multiprocessing module generates the process by creating a Process object and then calling its start () method, Process is the same as threading.Thread API.

Syntax format: multiprocessing.Process (group=None, target=None, name=None, args= (), kwargs= {}, *, daemon=None)

Parameter description:

Group: specifies the process group, which is not needed in most cases

Target: if you pass a reference to the function, you can task the child process to execute the code here

Name: set a name for the process, but not set it.

Args: parameters passed to the function specified by target, passed as tuples

Kwargs: pass named parameters to the function specified by target

The multiprocessing.Process object has the following methods and properties:

Method name / attribute description method start () Promoter process instance (create child process) join ([timeout]) executed by the run () process. If the optional parameter timeout is the default value None, it will block until the process calling the join () method is terminated. If timeout is a positive number, at most timeout seconds will block the alias of the current process. The default is that Process-N,N is an integer incremented from 1. The pid (process number) is_alive () of the current process pid determines whether the process child process is still alive, the exit code of the exitcode child process daemon process daemon, is a Boolean value. Authentication key for the authkey process. The numeric handle to the sentinel system object becomes ready when the process ends. Terminate () immediately terminates the child process kill () is the same as terminate (), regardless of whether the task is completed or not, but uses the SIGKILL signal on Unix. Close () closes the Process object Release all resources associated with it 2. 2 while loops execute together #-*-coding:utf-8-*-from multiprocessing import Processimport timedef run_proc (): "Code to be executed by child processes" while True: print ("--2 while name__=='__main__': -") time.sleep (1) if _ _ name__=='__main__': p = Process (target=run_proc) ) p.start () while True: print ("- 1 Murray Murray -") time.sleep (1)

Running result:

Note: when creating a child process, you only need to pass in a parameter that executes the function and function, create a Process instance, and start it with the start () method

3. Process pid#-*-coding:utf-8-*-from multiprocessing import Processimport osimport timedef run_proc (): "the code to be executed by the child process" print ('the child process is running " Pid=%d...'% os.getpid () # os.getpid gets the process number of the current process print ('the child process is about to end...') if _ _ name__ ='_ _ main__': print ('parent process pid:% d'% os.getpid ()) # os.getpid gets the process number of the current process p = Process (target=run_proc) p.start ()

Running result:

4. Pass the parameter #-*-coding:utf-8-*-from multiprocessing import Processimport osfrom time import sleepdef run_proc (name, age, * * kwargs) to the function specified by the child process: for i in range (10): print ('child process is running Name=% s name__=='__main__':% d, pid=%d...'% (name, age, os.getpid ()) print (kwargs) sleep (0.2) if _ _ name__=='__main__': p = Process (target=run_proc, args= ('test',18), kwargs= {"m": 20}) p.start () sleep (1) # 1 seconds later Immediately wrap up the child process p.terminate () p.join ()

Running result:

5. Global variable #-*-coding:utf-8-*-from multiprocessing import Processimport osimport timenums = [11,22] def work1 (): "" Code to be executed by child processes "" print ("in process1 pid=%d, nums=%s"% (os.getpid (), nums)) for i in range (3): nums.append (I) time.sleep (1) print ("in process1 pid=%d) Nums=%s "% (os.getpid (), nums) def work2 ():" Code to be executed by child process "" print ("in process2 pid=%d, nums=%s"% (os.getpid (), nums)) if _ _ name__ ='_ main__': p1 = Process (target=work1) p1.start () p1.join () p2 = Process (target=work2) p2.start ()

Running result:

In process1 pid=11349, nums= [11,22] in process1 pid=11349, nums= [11,22,0] in process1 pid=11349, nums= [11,22,0,1] in process1 pid=11349, nums= [11,22,0,1,2] in process2 pid=11350, nums= [11,22] III, Inter-process synchronization-Queue

Communication between Process is sometimes needed, and the operating system provides many mechanisms to realize the communication between processes.

1. Queue class syntax description method name indicates that q=Queue () initializes the Queue () object. If the maximum number of messages that can be received is not specified in parentheses, or if the number is negative, then there is no upper limit on the number of messages that can be accepted (until the end of memory) Queue.qsize () returns the number of messages contained in the current queue Queue.empty () if the queue is empty, return True, otherwise FalseQueue.full () if the queue is full Return True, or FalseQueue.get ([block [, timeout]]) gets a message from the queue and removes it from the queue, which defaults to True. 1. If block uses the default value and does not set timeout (in seconds), if the message queue is empty, the program will be blocked (stopped in the read state) until it is read from the message queue to the message. If timeout is set, it will wait for timeout seconds. If no message has been read, a "Queue.Empty" exception will be thrown. 2. If the block value is False, if the message queue is empty, it will immediately throw a "Queue.Empty" exception Queue.get_nowait () equivalent to Queue.get (False) Queue.put (item, [block [, timeout]]) to write the item message to the queue. The default value is True. 1. If block uses the default value and does not set timeout (in seconds), if there is no room to write in the message queue, the program will be blocked (stop in the write state) until space is made free from the message queue. If timeout is set, it will wait for timeout seconds. If there is no space, a "Queue.Full" exception will be thrown. 2. If the block value is False, if the message queue has no space to write, it will immediately throw a "Queue.Full" exception Queue.put_nowait (item) equivalent to Queue.put (item, False) 2. Use of Queue

You can use the Queue of the multiprocessing module to transfer data between multiple processes. Queue itself is a message queuing program. First, a small example is used to demonstrate how Queue works:

# coding=utf-8from multiprocessing import Queueq=Queue (3) # initialize a Queue object that can receive up to three put messages q.put ("message 1") q.put ("message 2") print (q.full ()) # Falseq.put ("message 3") print (q.full ()) # True# because the try below will throw an exception because the message queue is full. The first try will wait 2 seconds before throwing an exception. The second Try immediately throws an exception try: q.put ("message 4", True,2) except: print ("message queue is full, number of existing messages:% s"% q.qsize () try: q.put_nowait ("message 4") except: print ("message queue is full, number of existing messages:% s"% q.qsize ()) # recommended way to determine whether the message queue is full Then write if not q.full (): q.put_nowait ("message 4") # when reading a message, first determine whether the message queue is empty, and then read if not q.empty (): for i in range (q.qsize ()): print (q.get_nowait ())

Running result:

FalseTrue message queue is full, the number of existing messages: 3 message queue is full, the number of existing messages: 3 message 1 message 2 message 33. Queue instance

Let's take Queue as an example. Create two child processes in the parent process, one to write data to Queue and the other to read data from Queue:

From multiprocessing import Process, Queueimport os, time, random# write the code executed by the data process: def write (Q): for value in ['Aids,' B' 'C']: print (' Put% s to queue...'% value) q.put (value) time.sleep (random.random ()) # Code executed by the read data process: def read (Q): while True: if not q.empty (): value = q.get (True) print ('Get% s from queue.'% value) Time.sleep (random.random ()) else: breakif _ _ name__=='__main__': # parent process creates Queue And pass it to each child process: Q = Queue () pw = Process (target=write, args= (Q,)) pr = Process (target=read, args= (Q,)) # Promoter process pw, write: pw.start () # wait for pw to end: pw.join () # Promoter process pr, read: pr.start () pr.join () # pr process is an endless loop and cannot wait for it to end Can only be forcibly terminated: print ('') print ('all data is written and read')

Running result:

IV. Inter-process synchronization-Lock

The purpose of locking is to ensure data consistency. For example, read-write lock, each process adds 1 to a variable, but if another process reads and writes the value at the same time, the last written value is wrong. At this point, locks are needed to maintain data consistency.

Use Lock to control that a piece of code can only be executed by one process at a time. The two methods of the Lock object, acquire () are used to acquire the lock, and release () is used to release the lock. When a process calls acquire (), if the state of the lock is unlocked, it immediately changes to locked and returns, and the process acquires the lock. If the state of the lock is locked, the process that calls acquire () blocks.

1. Syntax description of Lock:

Lock = multiprocessing.Lock (): create a lock

Lock.acquire (): acquire the lock

Lock.release (): release the lock

With lock: automatically acquire and release locks similar to with open () as f:

two。 When the program is unlocked:

Import multiprocessingimport timedef add (num, value): print ('add {0}: num= {1}'. Format (value, num)) for i in range (0,2): num + = value print ('add {0}: num= {1}' .format (value, num)) time.sleep (1) if _ name__ = ='_ main__': lock = multiprocessing.Lock () num= 0 P1 = multiprocessing.Process (target=add, args= (num) 1)) p2 = multiprocessing.Process (target=add, args= (num, 2)) p1.start () p2.start ()

Running result: out of order, the two processes run alternately

Add1:num=0add1:num=1add2:num=0add2:num=2add1:num=2add2:num=4

3. When the program is locked:

Import multiprocessingimport timedef add (num, value, lock): try: lock.acquire () print ('add {0}: num= {1}' .format (value, num)) for i in range (0,2): num + = value print ('add {0}: num= {1}' .format (value) Num) time.sleep (1) except Exception as err: raise err finally: lock.release () if _ _ name__ = ='_ main__': lock = multiprocessing.Lock () num = 0p1 = multiprocessing.Process (target=add, args= (num, 1, lock)) p2 = multiprocessing.Process (target=add, args= (num, 2, lock) p1.start () p2.start ()

Run result: only when one of the processes is finished, the other processes will execute, and the one who grabs the lock first will execute first.

Add1:num=0add1:num=1add1:num=2add2:num=0add2:num=2add2:num=4 5. Process pool Pool

When the number of child processes to be created is small, you can directly use Process in multiprocessing to generate multiple processes dynamically, but if there are hundreds or even thousands of goals, it is a huge workload to create processes manually, so you can use the Pool method provided by the multiprocessing module.

1. Syntax description of Pool class

Syntax format: multiprocessing.pool.Pool ([processes [, initializer [, initargs [, maxtasksperchild [, context]))

Parameter description:

Processes: the number of worker processes. If processes is None, the value returned by os.cpu_count () is used.

Initializer: if initializer is not None, each worker process will call initializer (* initargs) at startup.

Maxtasksperchild: the number of tasks that a worker process can complete before it exits or is replaced by a new worker process, in order to release unused resources.

Context: used to specify the context of the worker process that was started.

There are two ways to submit tasks to the process pool:

Apply (func [, args [, kwds]]): blocking mode.

Apply_async (func [, args [, kwds]]): non-blocking mode. Call func using non-blocking mode (parallel execution, blocking mode must wait for the previous process to exit before executing the next process). Args is the parameter list passed to func, and kwds is the keyword parameter list passed to func.

Common functions of multiprocessing.Pool:

The method name states that close () closes Pool so that it no longer accepts new tasks terminate (), whether the task is completed or not, immediately terminates the blocking of the main process of join (), and waits for the exit of the child process. The instance of 2. Pool must be used after close or terminate

When initializing Pool, you can specify a maximum number of processes. When a new request is submitted to the Pool, if the pool is not full, a new process will be created to execute the request. However, if the number of processes in the pool has reached the specified maximum, the request will wait until there are processes in the pool. The previous process will not be used to execute the new task until the pool is finished. See the following example:

#-*-coding:utf-8-*-from multiprocessing import Poolimport os, time, randomdef worker (msg): t_start = time.time () print ("% s starts execution, process number is% d"% (msg,os.getpid ()) # random.random () randomly generates floating point numbers time.sleep (random.random () * 2) t_stop = time.time () print (msg, "execution complete Time-consuming% 0.2f "% (t_stop-t_start)) po = Pool (3) # define a process pool Maximum number of processes 3for i in range (0J10): # Pool (). Apply_async (target to be called, (parameter passed to target meta-ancestor,)) # each cycle will use idle child processes to call target po.apply_async (worker, (I,)) print ("- start----") po.close () # close the process pool After closing, po will no longer receive new requests po.join () # waiting for all child processes in po to complete execution, you must put print ("- end-") after the close statement.

Running result:

-start----0 starts execution, process number 214661 starts execution, process number 214682 starts execution, process number 214670 completes execution, takes 1.013 to start execution, process number 214662 completes execution, takes 1.244 to start execution, process number 214673 completes execution, takes 0.565 to start execution, process number 214661 completes execution, takes 1.686 to start execution, and process number 214684 completes execution. It takes 0.677 to start execution, process number 214675 to complete execution, 0.838 time to start execution, process number 214666 to complete execution, 0.759 to start execution, process number 214687 to complete execution, 1.038 to complete execution, 1.059 to complete execution, time-consuming 1.69-end-3. Queue in the process pool

If you want to create a process using Pool, you need to use Queue () in multiprocessing.Manager ()

Instead of multiprocessing.Queue (), you will get the following error message: RuntimeError: Queue objects should only be shared between processes through inheritance.

The following example demonstrates how processes in a process pool communicate:

#-*-coding:utf-8-*-# modify Queue in import to Managerfrom multiprocessing import Manager,Poolimport os,time,randomdef reader (Q): print ("reader startup (% s), parent process (% s)"% (os.getpid ()) Os.getppid ()) for i in range (q.qsize ()): print ("reader got message from Queue:% s"% q.get (True)) def writer (Q): print ("writer startup (% s), parent process (% s)"% (os.getpid ()) Os.getppid ()) for i in "itcast": q.put (I) if _ _ name__== "_ _ main__": print ("(% s) start"% os.getpid ()) Q = Manager (). Queue () # use Queue po = Pool () po.apply_async (writer, (Q,)) time.sleep (1) # Let the above task store data in Queue first Then let the following task start fetching data from po.apply_async (reader, (Q,)) po.close () po.join () print ("(% s) End"% os.getpid ())

Running result:

(11095) startwriter startup (11097), parent process (11095) reader startup (11098), parent process (11095) reader get messages from Queue: ireader gets messages from Queue: treader gets messages from Queue: creader gets messages from Queue: areader gets messages from Queue: sreader gets messages from Queue: t (11095) end 6, process and thread comparison 1. Function

Process: ability to perform multiple tasks, such as the ability to run multiple QQ simultaneously on a single computer

Threads: ability to perform multiple tasks, such as multiple chat windows in a QQ

Different definitions

The process is an independent unit for resource allocation and scheduling in the system.

A thread is an entity of a process and the basic unit of CPU scheduling and dispatching. It is a smaller basic unit that can run independently than a process. The thread basically does not own the system resources, but only has some essential resources (such as program counters, a set of registers and stacks), but it can share all the resources owned by the process with other threads belonging to the same process.

two。 Difference

A program has at least one process and a process has at least one thread.

-the division scale of threads is smaller than that of processes (fewer resources than processes), which makes multithreaded programs have high concurrency.

-the process has an independent memory unit during execution, and multiple threads share memory, thus greatly improving the running efficiency of the program

Thread cannot be executed independently and must be stored in the process.

You can think of a process as an assembly line in a factory, and the thread in it is the worker on the assembly line.

3. Advantages and disadvantages

Thread: thread execution overhead is small, but it is not conducive to resource management and protection.

Process: the execution cost of the process is high, but it is beneficial to the management and protection of resources.

Thank you for your reading, these are the contents of "Python multi-process knowledge points". After the study of this article, I believe you have a deeper understanding of what Python multi-process knowledge points have, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.