Shulou(Shulou.com)06/01 Report--

This article mainly introduces the relevant knowledge of "how python realizes multi-process concurrency control Semaphore and mutex LOCK". The editor shows you the operation process through an actual case, and the operation method is simple, fast and practical. I hope that this article "how to achieve multi-process concurrency control Semaphore and mutex LOCK" in python can help you solve the problem.

First, understand the lock

Application scenario description: Lock mutex: for example-there are three colleagues who need to go to the toilet at the same time, but there is only one toilet, compare colleagues to processes, multiple processes and play to seize a toilet, we want to ensure that the order is priority, one by one, then it must be serial, first come first, someone will use it, lock it, and the rest will continue to compete after the end.

1. Use Lock to deal with

Simulate three colleagues to seize the toilet

From multiprocessing import Processfrom multiprocessing import Lockimport timeimport randomdef task1 (p, lock): # Lock lock.acquire () print (f'{p} start excretion') time.sleep (random.randint (1,3)) print (f'{p} end of excretion') # unlock lock.release () def task2 (p, lock): lock.acquire () print (f'{p} start excretion') time.sleep (random.randint (1) 3) print (f'{p} end of excretion') lock.release () def task3 (p, lock): lock.acquire () print (f'{p} start excretion') time.sleep (random.randint (1) 3) print (f'{p} end of excretion') lock.release () if _ _ name__ = ='_ _ main__': # instantiate a lock object mutex = Lock () # pass the lock into the process object p1 = Process (target=task1, args= ('task1', mutex,)) p2 = Process (target=task2, args= (' task2', mutex,)) p3 = Process (target=task3, args= ('task3', mutex) ) p1.start () p2.start () p3.start ()

Execution result:

# output result: the three processes begin to scramble for mutexes. Those that are grabbed first are executed first. After the execution is finished, the locks are released, and the rest continue to compete.

Task1 begins to excrete.

End of task1 excretion

Task2 begins to excrete.

End of task2 excretion

Task3 begins to excrete.

End of task3 excretion

1. Note:

The mutex is locked in the function, and every time lock.acquire () is locked, lock.release () is unlocked, and the code that needs to be executed is written between locking and unlocking.

If you lock it more than twice in a row, a deadlock will occur and the code will not continue to execute. The lock must be unlocked one at a time.

2. Lock and join comparison:

Common ground-can turn parallel into serial, ensuring the order of execution

Differences-join sets the order artificially, while lock allows it to compete in order to ensure fairness.

2. Use reflection to optimize the above code

Although the above code plays a first-in, first-out, one-in-one effect, but it is not perfect. It is well known that whoever gets the first to go to the bathroom does not follow the order of start () in the code. It is more reasonable to be limited by the process that is preempted.

From multiprocessing import Processfrom multiprocessing import Lockimport timeimport randomimport sysdef task1 (p, lock): # Lock lock.acquire () print (f' {p} start printing') time.sleep (random.randint (1,3)) print (f' {p} print end') # unlock lock.release () def task2 (p Lock): lock.acquire () print (f'{p} start printing') time.sleep (random.randint (1,3)) print (f'{p} print end') lock.release () def task3 (p, lock): lock.acquire () print (f'{p} start printing') time.sleep (random.randint (1) 3) print (f' {p} print end') lock.release () if _ _ name__ ='_ _ main__': slock = Lock () for i in range (1 name__ 4): P = Process (target=getattr (sys. Modules [_ _ name__], f'task {I}'), args= (f'task {I}', slock)) p.start ()

Output result:

Task2 starts printing

End of task2 print

Task3 starts printing

End of task3 print

Task1 starts printing

End of task1 print

2. Process concurrency control semaphore

Semaphore (semaphore): used to control the number of visits to shared resources, and can control the number of concurrent processes at the same time.

Semaphore: it is also a lock, but it does not guarantee data security, and multiple threads are opened at the same time, but it specifies the upper limit of concurrent execution at the same time. (to control the number of concurrency)

1. Example of multi-process control (1)

# example: there are 5 potholes in a toilet, and a maximum of 5 people can go to the toilet at the same time. At present, 20 people want to go to the toilet, but only 5 people can go in, and then how many can come out? how many people can go to the toilet # when multiple functions are invoked from one module Separate from threading import Semaphore, Thread, currentThreadimport timeimport randomsem = Semaphore (3) # concurrent execution with commas set to 5def task (): sem.acquire () print (f'{currentThread (). Name}') time.sleep (random.randint (1Magne3)) sem.release () if _ name__ = ='_ main__': for i in range (20): t = Thread (target=task) t.start ()

Execution result: the first concurrency is 3, and the lock is executed first.

Thread-1

Thread-2

Thread-3

Thread-4

Thread-5

Thread-6

Thread-7

Thread-8

Process finished with exit code 0

two。 Example of multi-process control (2)

Import multiprocessingimport timedef worker (s, I): s.acquire () print (time.strftime ('% Y-%m-%d% HGV% MVR% S'), multiprocessing.current_process (). Name + "preempt and acquire lock, run") time.sleep (I) print (time.strftime ('% Y-%m-%d% HRV% MRV% S'), multiprocessing.current_process (). Name + "end of run" Release lock ") s.release () if _ _ name__ ='_ _ main__': s = multiprocessing.Semaphore (2) for i in range (8): P = multiprocessing.Process (target=worker, args= (s, 1)) p.start ()

Execution result:

In the terminal where the output of the execution result is blocked, press the enter key each time, the concurrent execution of the process can be seen more clearly.

As can be seen from the execution results below, there are two processes executing at the same time.

2021-05-18 22:50:37 Process-1 preempted and acquired the lock, run

2021-05-18 22:50:37 Process-2 preempted and acquired the lock, run

2021-05-18 22:50:38 Process-1 operation ends, release lock

2021-05-18 22:50:38 Process-3 preempted and acquired the lock, run

2021-05-18 22:50:38 Process-2 operation ends, release lock

2021-05-18 22:50:38 Process-4 preempted and acquired the lock, run

2021-05-18 22:50:39 Process-3 operation ends, release lock

2021-05-18 22:50:39 Process-5 preempted and acquired the lock, run

2021-05-18 22:50:39 Process-4 operation ends, release lock

2021-05-18 22:50:39 Process-6 preempted and acquired the lock, run

2021-05-18 22:50:40 Process-5 operation ends, release lock

2021-05-18 22:50:40 Process-7 preempted and acquired the lock, run

2021-05-18 22:50:40 Process-6 operation ends, release lock

2021-05-18 22:50:40 Process-8 preempted and acquired the lock, run

2021-05-18 22:50:41 Process-7 operation ends, release lock

2021-05-18 22:50:41 Process-8 operation ends, release lock

Process finished with exit code 0

3. LOCK of process synchronization

Multiple processes execute concurrently to improve resource utilization and efficiency, but sometimes we need to use lock LOCK if only one process can access a shared resource at a given time.

1. Example of not adding LOCK

Import multiprocessingimport timedef task1 (): n = 4 while n > 1: print (f'{time.strftime ("% Y-%M-%d% H:%M:%S")} task1 output information') time.sleep (1) n-= 1def task2 (): n = 4 while n > 1: print (f'{time.strftime ("% Y-%M-%d% HJV% M: % S ")} task2 output information') time.sleep (1) n-= 1def task3 (): n = 4 while n > 1: print (f'{time.strftime ("% Y-%M-%d% H:%M:%S ")} task3 output information') time.sleep (1) n-= 1if _ name__ = ='_ main__': P1 = multiprocessing.Process (target=task1) p2 = multiprocessing.Process (target=task2) p3 = multiprocessing.Process (target=task3) p1.start () p2.start () p3.start ()

Execution result:

2021-59-18 22:59:46 task1 output information

2021-59-18 22:59:46 task2 output information

2021-59-18 22:59:46 task3 output information

2021-59-18 22:59:47 task1 output information

2021-59-18 22:59:47 task2 output information

2021-59-18 22:59:47 task3 output information

2021-59-18 22:59:48 task1 output information

2021-59-18 22:59:48 task2 output information

2021-59-18 22:59:48 task3 output information

Process finished with exit code 0

two。 Add the example of LOCK

There are two ways to lock: first, generate the lock object lock from lock = multiprocessing.Lock ()

With lock: with starts lock before execution and closes lock after execution

Lock.acquire ()... Lock.release (): note that these two must correspond one after another.

Import multiprocessingimport timedef task1 (lock): with lock: n = 4 while n > 1: print (f'{time.strftime ("% Y-%M-%d% H:%M:%S")} task1 output information') time.sleep (1) n-= 1def task2 (lock): lock.acquire () n = 4 while n > 1: Print (f'{time.strftime ("% Y-%M-%d% H:%M:%S")} task2 output information') time.sleep (1) n-= 1 lock.release () def task3 (lock): lock.acquire () n = 4 while n > 1: print (f'{time.strftime ("% Y-%M-%d% H:%M:%S")} task3 Output information') time.sleep (1) n-= 1 lock.release () if _ _ name__ = ='_ main__': lock = multiprocessing.Lock () p1 = multiprocessing.Process (target=task1 Args= (lock,) p2 = multiprocessing.Process (target=task2, args= (lock,)) p3 = multiprocessing.Process (target=task3, args= (lock,)) p1.start () p2.start () p3.start ()

Execution result

2021-11-18 23:11:37 task1 output information

2021-11-18 23:11:38 task1 output information

2021-11-18 23:11:39 task1 output information

2021-11-18 23:11:40 task2 output information

2021-11-18 23:11:41 task2 output information

2021-11-18 23:11:42 task2 output information

2021-11-18 23:11:43 task3 output information

2021-11-18 23:11:44 task3 output information

2021-11-18 23:11:45 task3 output information

Process finished with exit code 0

This is the end of the introduction on "how python implements multi-process concurrency control Semaphore and mutex LOCK". Thank you for reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.