Shulou(Shulou.com)06/03 Report--

This article mainly introduces "Event events, process pools, thread pools and coroutines in python". In daily operation, I believe many people have doubts about Event events, process pools, thread pools and coroutines in python. Xiaobian consulted all kinds of information and sorted out simple and easy to use operation methods. I hope to answer your doubts about "Event events, process pools, thread pools and coroutines in python"! Next, please follow the small series to learn together!

Event Events

Used to control thread execution

e.wait(), this thread will be set to False, can not perform this task;

As soon as e.set() appears in a thread, it tells the Event object to change all users with e.wait to True, and the rest of the tasks will be executed immediately. Some threads control other threads, with events in between.

from threading import Eventfrom threading import Threadimport time#Call Event instantiate object e = Event()## #False if method appears in task, block # e.wait() # False# #If method appears in task, change False for other threads to True, enter ready and runtime # e.set() # Truedef light(): print ('red light on... ') time.sleep(5) #should signal to other threads that they are ready to execute e.set() #change False in car to True print ('green light... ')def car(name): print ('waiting for red light... ') #Put all car tasks into blocking state e.wait() # False print(f'{name} is accelerating... ')#Let a light thread control multiple car threads t = Thread(target=light)t.start()for i in range(10): t = Thread(target=car, args=(f' car {i} number',)) t.start()

Process pool and thread pool

Process pools and thread pools are used to control the number of processes/threads allowed by the current program

Function: Ensure that the number of processes/threads created is within the hardware's allowable range

Thread pool usage one:

from concurrent.futures import ThreadPoolExecutorimport timepool = ThreadPoolExecutor(5) # 5 represents only 5 processes can be started, without default CPU # ThreadPoolExecutor(5) # 5 represents only 5 threads can be started # pool.submit() #asynchronous submit task, parenthesis function address def task(): print ('thread task started... ') time.sleep(1) print (') Thread task finished... ')for line in range(5): pool.submit(task)

Use 2:

from concurrent.futures import ThreadPoolExecutorimport timepool = ThreadPoolExecutor(5) # 5 represents only 5 processes can be started, without default CPU # ThreadPoolExecutor(5) # 5 represents only 5 threads can be started # pool.submit() #asynchronous submit task, parenthesis function address def task(): print ('thread task started... ') time.sleep(1) print (') Thread task finished... ') return 123#callback function def call_back(res): print(type(res)) res2 = res.result() #note: assignment operation does not have the same name as res received print(res2)for line in range(5): pool.submit(task).add_done_callback(call_back)

pool.shutdown() causes all thread pool tasks to finish before proceeding to code execution

Multi-thread pear crawling video

Encapsulate the underlying socket socket with the requests module

Get all video id numbers in the home page, splice video details page url

Get real videos in video details url srcUrl=

Send a request to get video binary data to the real video url address

Finally, save the video binary data locally

coroutine

Process: Resource Unit

Thread: Unit of Execution

Coroutine: Concurrency in a single thread

Note: coroutines are not operating system resources, the purpose is to allow single threads to achieve concurrency

goal of coordination

Operating system: use multi-channel technology, switch + save state, one is IO encountered, the other is CPU execution time is too long

Protocol: switch + save state by manually simulating operating system "multi-channel counting"

Manual implementation, encounter IO switch, trick the operating system into thinking there is no IO operation

Single thread, IO encountered, switch + save state

Single thread, for computationally intensive, switching back and forth + save state is less efficient

Benefits: Increased efficiency in IO-intensive situations

Disadvantages: If you switch back and forth in a computationally intensive situation, it will be less efficient.

import timedef func1(): for i in range (10000000): i+1def func2(): for i in range (10000000): i+1start = time.time()func1()func2()stop = time.time()print (stop - start) # 1.0312113761901855#yield-based concurrency Less efficient in computationally intensive cases def func1(): while True: 1000000 +1 yielddef func2(): g = func1() for i in range (1000000): i+1 next(g) #Each execution of next corresponds to switching to func1 below start = time.time()func2()stop = time.time()print(stop-start) # 1.3294126987457275

gevent

gevent is a third-party module that can help you monitor IO operations and switch

The purpose of using gevent: to be implemented under single thread, save state + switch when IO is encountered

import timefrom gevent import monkeymonkey.patch_all() #can listen to all IO operations under this program from gevent import spawn, joinall #used to switch + save state def func1(): print ('1 ') time.sleep(1) # IO operations def func2(): print ('2') time.sleep(3)def func3(): print ('3 ') time.sleep(5)start = time.time()s1 = spawn(func1)s2 = spawn(func2)s3 = spawn(func3)s1.join() #Send a signal, equivalent to waiting for yourself (In case of single thread) s2.join()s3.join()# joinall((s1, s2, s3)) #It is troublesome to execute them one by one. You can use joinall to load them all end = time.time()print(end-start) # 5.006161451339722

TCP socket socket implementation protocol

Server:

from gevent import monkeyfrom gevent import spawnimport socketmonkey.patch_all()server = socket.socket()server.bind(('127.0.0.1', 9999))server.listen(5)def task(conn): while True: try: data = conn.recv(1024) if len(data) == 0: break print(data.decode('utf-8')) send_data = data.upper() conn.send(send_data) except Exception: break conn.close()def server2(): while True: conn, addr = server.accept() print(addr) spawn(task, conn)if __name__ == '__main__': s = spawn(server2) s.join()

Client:

import socketfrom threading import Thread, current_threaddef client(): client = socket.socket() client.connect (('127.0.0.1', 9999)) number = 0 while True: send_data = f'{current_thread().name} {number}' client.send (send_data.encode('utf-8')) data = client.recv(1024) print (data.decode ('utf-8')) number += 1 for i in range(400): t = Thread(target=client) t.start() At this point, the study of "Event events, process pools, thread pools and coroutines in python" is over, hoping to solve everyone's doubts. Theory and practice can better match to help everyone learn, go and try it! If you want to continue learning more relevant knowledge, please continue to pay attention to the website, Xiaobian will continue to strive to bring more practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.