Shulou(Shulou.com)06/03 Report--

This article mainly explains "how to understand python yeild,async,azwait and coroutine", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's learn how to understand Python's yeild,async,azwait and coroutine.

directory

1. Analysis of yield usage

Use of async and await

1. What are processes, coroutines, asynchronous?

2. How to handle 200W URLs and save all URLs?

3. await and gather using async

III. Understanding of coordination

1. Analysis of yield usage

yield is a generator that returns an interable object.

This object has a next() method, which lets you see what the next element is.

1. Interchangeable objects, objects that can be traversed, such as: list, str, tuple, dict, file, xrange, etc.

2 What is the function of yield? Just a way to get intermediate variables inside the loop, save the desired variables each time you use yield until the end of the loop, and get a generator object at the end of the loop.

3. Why use yield? Using yield, a function rewritten as a generator has the ability to iterate. Compared to using the instance of the class to save the state to calculate the next value that needs iteration, the code is simpler and the execution process is very simple.

4. How to determine the type of yield?

def fab(max): n, a, b = 0, 0, 1 while n

< max: yield b # 使用 yield # print b a, b = b, a + b n = n + 1for n in fab(5): print n fab不是generator，fab(5)是generator。 好比类的定义和类的实例的区别。 >

>>import types >>> isinstance(fab, types.GeneratorType) False >>> isinstance(fab(5), types.GeneratorType) True

Fab is non-iterable, and fab(5) is iterable.

>>>from collections import Iterable >>> isinstance(fab, Iterable) False >>> isinstance(fab(5), Iterable) True

5. Yield in file reading applications?

If bytes read a file using read(), this results in an unpredictable memory footprint. A good approach is to use yield, a fixed-length buffer that reads files over and over, to generate an iterative generator that reads files.

def read_file(fpath): BLOCK_SIZE = 1024 with open(fpath, 'rb') as f: while True: block = f.read(BLOCK_SIZE) if block: yield block else: return 2. Use of async and await 1. What is a process, coroutine, asynchronous?

What's a protocol?

A user-level lightweight thread that has its own register context and stack.

During coroutine switching, registers and stacks are saved elsewhere, and when returned, the saved register context and stack are restored.

Why use coroutines?

Mainstream languages use multithreaded concurrency, thread-dependent concepts are preemptive multitasking, coroutine-dependent collaborative multitasking.

Whether multi-process or multi-threaded, each block, switch gets stuck in a system call.

The CPU runs the operating system's scheduler, which decides which process (thread) to run.

Threads handle synchronization issues very carefully, whereas coroutines have no such issues at all.

For CPU, multi-coroutines are single-threaded, CPU does not consider scheduling, switching context, saving CPU switching overhead. Why coroutines are better than multithreading.

How do I use coroutines?

Under multi-process + coroutine, it avoids the overhead of CPU switching, and can make full use of multiple CPUs. This method is very efficient for crawlers with large data volume and file reading and writing.

2. How to handle 200W URLs and save all URLs?

Single process + single thread

Single process + multithreading: open ten threads, speed can not be increased tenfold. Thread switching is costly, and there is no way to create threads indefinitely.

Multi-process + multi-threading: Each process of multi-process consumes a CPU, and multi-threading bypasses blocking time to a certain extent, so multi-threading is more efficient than single-process multi-threading.

coroutine

3. await and gather using async

await accepts a coroutine list and returns done and pending lists. done is a completed coroutine, pending is a coroutine that is still running. Get the completed result via.result()

Gather accepts the coroutine as gather(cro1, cro2, cro3, cro4…) and returns a coroutine that combines so many tasks.

Use of async: blog.csdn.net/qq_29785317/article/details/103294235

async def func1(num): print('--func1 start--') await asyncio.sleep(num) print('--func1 done--') return 'func1 ok'async def func2(num): print('--func2 start--') await asyncio.sleep(num) print('--func2 done--') return 'func2 ok'async def main(): task1 = asyncio.ensure_future(func1(3)) task2 = asyncio.ensure_future(func2(5)) tasks = [task1, task2] res = await asyncio.gather(*tasks) return res # done, pending = await asyncio.wait(tasks) # for t in done: # print(t.result()) # print(done) # print(pending)if __name__ == '__main__': loop = asyncio.get_event_loop() result = loop.run_until_complete(main()) print(result)```python--func1 start---func2 start---func1 done--func2 done--['func1 ok','func2 ok'] III. Understanding of coroutine

1. coprocedural process

Yield in a coroutine is the way the flow is controlled.

Yield, like receiver, is a generator that needs to be activated before it can be used.

>>> def simple_corotine():... print('---->coroutine started')... x = yield #has a receive value, so like the generator, you need to activate it first, use next... print('---->coroutine recvied:',x)...>>> my_coro = simple_corotine()>>> my_coro>> next(my_coro) #Activate the generator first, execute to yield val statement #or use send(None) to activate the generator--->coroutine started>> my_coro.send(24) #Pass the value into it, x = yield--->coroutine revived: 24Traceback (most recent call last): File "", line 1, in StopIteration #An error is reported when generator execution is complete

2. Four states of coroutine in operation

GEN_CREATE: Waiting to start execution

GEN_RUNNING: Interpreter is executing, this status is generally invisible

GEN_SUSPENDED: pause at yield expression

GEN_CLOSED: End of execution

>>> def averager():... total = 0.0... count = 0... aver = None... while True:... term = yield aver... total += term... count += 1... aver = total/count...>>> coro_avg = averager()>>> coro_avg.send(None)>>> coro_avg.send(10)10.0>>> coro_avg.send(20)15.0>>> coro_avg.send(30)20.0>>> coro_avg.send(40)25.0

At the end of each loop pause at yield until the next parameter is passed in.

3. Decorators for pre-activation routines (custom activation)

What is the function of @ Decorator? Decorate the original function, adding a new function and way to the crude function.

Why does @ work as a decorator? Functions are also objects, and functions can be passed to pinch functions as arguments.

>>> def coro_active(func):... def inner(*args,**kwargs):... gen = func(*args,**kwargs)... next(gen) #gen.send(None)... return gen... return inner...>>> @coro_active... def averager():... total = 0.0... count = 0... aver = None... while True:... term = yield aver... total += term... count += 1... aver = total/count...>>> coro_avg = averager()>>> coro_avg.send(10) 10.0 >>> coro_avg.send(20) 15.0 >>> coro_avg.send(30) 20.0

4. Terminate coroutines and exception handling

When the coroutine's next function or send function fails, the coroutine terminates.

An exception trap needs to be created to handle exceptions to the coroutine and close the current coroutine.

5. Let coroutine return value

How yield is used

At this point, I believe that everyone has a deeper understanding of "how to understand python yeild,async,azwait and coroutine", so let's actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.