Shulou(Shulou.com)05/31 Report--

Today, the editor will share with you the relevant knowledge points about how to use the python protocol and asyncio library. The content is detailed and the logic is clear. I believe most people still know too much about this knowledge, so share this article for your reference. I hope you can get something after reading this article. Let's take a look.

1.asyncio Asynchronous Icano Library

The asyncio library in python provides methods for managing events, collaborators, tasks, and threads, as well as primitives for writing concurrent code, namely async and await.

The main contents of this module are:

Event loop: event_loop, which manages all events, is an infinite loop method that tracks the order in which events occur during the loop, puts them in a queue, and calls the appropriate event handlers to handle them when idle

Coroutine: the generalized concept of a subroutine that pauses during execution, waits for the completion of external processing (Imax O operation), and then continues from the paused place. The function definition uses the async keyword, so that the function does not execute immediately, but returns a co-program object.

Future and Task:Future objects represent unfinished calculations. Task is a subclass of Future that contains the state of a task to run multiple tasks concurrently while running a task.

The definition of Asynchronous function

In essence, an asynchronous function is still a function, but it will hand over the execution power to other coprograms during execution, which is different from the ordinary function definition by adding async before the def keyword.

# Asynchronous function import asyncio# Asynchronous function async def func (x): print (Asynchronous function) return x * * 2ret = func (2) print (ret)

Run the code and enter the following:

Sys:1: RuntimeWarning: coroutine 'func' was never awaited

The function returns a collaborator object that needs to be placed in the event loop event_loop if you want the function to be executed.

Event Loop event_loop

Event_loop is the core of the asyncio module, which registers asynchronous functions with event loops. The process is implemented by loop calling the co-program at the appropriate time, which is called asyncio.get_event_loop (), and then the run_until_complete (the co-program object) registers the co-program into the event loop and starts the event loop.

Import asyncio# asynchronous function async def func (x): print ("asynchronous function") return x * * cooperative program object, which cannot directly run coroutine1 = func (2) # event loop object loop = asyncio.get_event_loop () # add the co-program object to the event loop and execute ret = loop.run_until_complete (coroutine1) print (ret)

The first step in using asynchronous functions in versions prior to python 3.7 is to install the above process:

First get the event loop loop object through asyncio.get_event_loop ()

The asynchronous function is then executed by calling loop.run_until_complete () or loop.run_forever () with different policies.

In versions later than python 3.7, you can simply use asyncio.run (), which always creates a new event loop and closes it at the end.

The latest official documentation uses the run approach. Official case

Import asyncioasync def main (): print ('hello') await asyncio.sleep (1) print (' world') asyncio.run (main ())

Next, take a look at a complete case, combined with the await keyword.

Import asyncioimport time# Asynchronous function 1async def task1 (x): print ("Task 1") await asyncio.sleep (2) print ("restore Task 1") return x # Asynchronous function 2async def task2 (x): print (Task 2) await asyncio.sleep (1) print ("restore Task 2") return xasync def main (): start_time = time.perf_counter () ret _ 1 = await task1 (1) ret_2 = await task2 (2) print ("the value returned by Task 1 is" Ret_1) print ("the value returned by Task 2 is", ret_2) print ("run time", time.perf_counter ()-start_time) if _ _ name__ = ='_ _ main__': # create an event loop loop = asyncio.get_event_loop () # add the collaborator object to the event loop And execute loop.run_until_complete (main ())

The code output is as follows:

Task 1

Restore Task 1

Task 2

Restore Task 2

The value returned by Task 1 is 1

The value returned by Task 2 is 2

Run time 2.99929154

The above code creates three coprograms, in which task1 and task2 are placed in the coprogram function main, and the Icano operation is simulated by asyncio.sleep (1). The running time of the whole function is 2.9999 seconds, nearly 3 seconds, which is still serial. If you want to change it to concurrent execution, modify the code as follows.

Import asyncioimport time# Asynchronous function 1async def task1 (x): print ("Task 1") await asyncio.sleep (2) print ("restore Task 1") return x # Asynchronous function 2async def task2 (x): print (Task 2) await asyncio.sleep (1) print ("restore Task 2") return xasync def main (): start_time = time.perf_counter () ret_1 Ret_2 = await asyncio.gather (task1 (1), task2 (2)) print ("the value returned by task 1 is", ret_1) print ("the value returned by task 2 is", ret_2) print ("run time", time.perf_counter ()-start_time) if _ _ name__ = ='_ main__': loop = asyncio.get_event_loop () loop.run_until_complete (main ())

The biggest change in the above code is that task1 and task2 are put into asyncio.gather () to run, and the output time of the code is significantly shorter.

Task 1

Task 2

Restore Task 2 # Task 2 returns first because of the short waiting time.

Restore Task 1

The value returned by Task 1 is 1

The value returned by Task 2 is 2

Run time 2.00056694800003

Asyncio.gather () can be replaced with asyncio.wait (), and the modified code is as follows:

Import asyncioimport time# Asynchronous function 1async def task1 (x): print ("Task 1") await asyncio.sleep (2) print ("restore Task 1") return x # Asynchronous function 2async def task2 (x): print (Task 2) await asyncio.sleep (1) print ("restore Task 2") return xasync def main (): start_time = time.perf_counter () done Pending = await asyncio.wait ([task1 (1), task2 (2)]) print (done) print (pending) print ("run time", time.perf_counter ()-start_time) if _ _ name__ ='_ main__': loop = asyncio.get_event_loop () loop.run_until_complete (main ())

Asyncio.wait () returns a tuple containing a collection of completed tasks and a collection of unfinished tasks.

The difference between gather and wait:

Gather: all tasks need to be finished. If any of the co-programming functions crashes, an exception will be thrown and no result will be returned.

Wait: you can define the time when the function returns, which can be set to FIRST_COMPLETED (the first to end), FIRST_EXCEPTION (the first exception occurs), and ALL_COMPLETED (all executed by default).

Done,pending = await asyncio.wait ([task1 (1), task2 (2)], return_when=asyncio.tasks.FIRST_EXCEPTION) create task

Because the collaborator object cannot be run directly, it is the run_until_complete method that wraps it as a task object when registered with the event loop. This object is a further encapsulation of the coroutine object, which has more running states than the coroutine object, such as pending,running,finished, which can be used to obtain the execution of the collaborator object.

The coroutine object shown below is encapsulated into a task object, which is modified based on the above code.

Import asyncioimport time# asynchronous function 1async def task1 (x): print ("Task 1") await asyncio.sleep (2) print ("restore Task 1") return x # Asynchronous function 2async def task2 (x): print ("Task 2") await asyncio.sleep (1) print ("restore Task 2") return xasync def main (): start_time = time.perf_counter () # Encapsulates the task object coroutine1 = task1 (1) task_1 = loop.create_task (coroutine1) coroutine2 = task2 (2) task_2 = loop.create_task (coroutine2) ret_1 Ret_2 = await asyncio.gather (task_1, task_2) print ("the value returned by task 1 is", ret_1) print ("the value returned by task 2 is", ret_2) print ("run time", time.perf_counter ()-start_time) if _ _ name__ ='_ _ main__': loop = asyncio.get_event_loop () loop.run_until_complete (main ())

Because the task object is a subclass object of the future object, the above code can also be modified as follows:

# task_2 = loop.create_task (coroutine2) task_2 = asyncio.ensure_future (coroutine2)

Let's print out the states of the task object.

Import asyncioimport time# asynchronous function 1async def task1 (x): print ("Task 1") await asyncio.sleep (2) print ("restore Task 1") return x # Asynchronous function 2async def task2 (x): print ("Task 2") await asyncio.sleep (1) print ("restore Task 2") return xasync def main (): start_time = time.perf_counter () # Encapsulate the task object coroutine1 = task1 (1) task_1 = loop.create_task (coroutine1) coroutine2 = task2 (2) # task_2 = loop.create_task (coroutine2) task_2 = asyncio.ensure_future (coroutine2) # enter pending state print (task_1) print (task_2) # get the completion status of the task print (task_1.done () Task_2.done ()) # execute task await task_1 await task_2 # get completion status again print (task_1.done (), task_2.done ()) # get the returned result print (task_1.result ()) print (task_2.result ()) print ("run time" Time.perf_counter ()-start_time) if _ _ name__ = ='_ main__': loop = asyncio.get_event_loop () loop.run_until_complete (main ())

Await task_1 represents the execution of the protocol, and after execution, task.done () returns True,task.result () to get the return value.

Callback return value

You need to get the return value of the cooperative program when it is finished. You have just demonstrated a way to use the task.result () method to get the result, but this method can only get the result when the collaborative program is finished. If the collaborative program does not finish running, the result () method will return asyncio.InvalidStateError (invalid status error).

Generally, the second scheme is used for coding, which binds the callback through the add_done_callback () method.

Import asyncioimport requestsasync def request_html (): url = 'https://www.csdn.net' res = requests.get (url) return res.status_codedef callback (task): print (' callback:' Task.result () loop = asyncio.get_event_loop () coroutine = request_html () task = loop.create_task (coroutine) # bind callback task.add_done_callback (callback) print (task) print ("*" * 100) loop.run_until_complete (task) print (task)

When the above code finishes executing coroutine, it calls the callback function.

If the callback function requires more than one argument, use the partial method in the functools module

Loop event off

It is recommended that after each coding, the loop event object close () method is called to thoroughly clean up the loop object.

two。 This section crawler project

The sites to be collected in this class are all coser images, so the address can be checked in the code.

The complete code is as follows:

Import threadingimport asyncioimport timeimport requestsimport lxmlfrom bs4 import BeautifulSoupasync def get (url): return requests.get (url) async def get_html (url): print ("ready to grab:", url) res = await get (url) return res.textasync def save_img (img_url): # thumbMid_5ae3e05fd3945 replace the small image with the larger image img_url = img_url.replace ('thumb') 'thumbMid') img_url = "http://mycoser.com/" + img_url print (" Image download: ", img_url) res = await get (img_url) if res is not None: with open (f'./imgs/ {time.time ()} .jpg', 'wb') as f: f.write (res.content) return img_url "ok" async def main (url_list): # create 5 tasks tasks = [asyncio.ensure_future (get_html (url_ list [_])) for _ in range (len (url_list))] dones, pending = await asyncio.wait (tasks) for task in dones: html = task.result () soup = BeautifulSoup (html 'lxml') divimg_tags = soup.find_all (attrs= {' class': 'workimage'}) for div in divimg_tags: ret = await save_img (div.a.img ["data-original"]) print (ret) if _ name__ = =' _ main__': urls = [f "http://mycoser.com/picture/lists/p/{page}" for page in range (1) 17)] totle_page = len (urls) / / 5 if len (urls)% 5 = = 0 else len (urls) / / 5 + 1 # slice the urls list Convenient collection of for page in range (0, totle_page): start_page = 0 if page = = 0 else page * 5 end_page = (page + 1) * 5 # cyclic event object loop = asyncio.get_event_loop () loop.run_until_complete (main (urls [start _ page:end_page]))

Code description: the first thing to note in the above code is that the await keyword can only be followed by the following:

Native co-program object

An iterator returned by an object that contains the await method.

So a co-program get is nested in the get_html function of the above code. In order to facilitate the operation in the main function main, the urls is sliced directly and then run through a loop.

Of course, the last two lines of the above code can be directly modified to:

# Loop event object # loop = asyncio.get_event_loop () # # loop.run_until_complete (main (urls [start _ page:end_page])) asyncio.run (main (urls [start _ page:end_page]))

Easy access to a bunch of high-definition pictures:

These are all the contents of this article entitled "how to use python and asyncio Library". Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.