Shulou(Shulou.com)05/31 Report--

In this article, the editor introduces in detail "how to solve the most powerful error retest database problem in Python". The content is detailed, the steps are clear, and the details are handled properly. I hope this article "how to solve the most powerful error retest library problem in Python" can help you solve your doubts.

1 introduction

When we write programs, especially those related to web requests, such as calling the web interface, running web crawlers and other tasks, we often encounter some occasional request failures. At this time, if we simply catch errors and skip the corresponding tasks, it is certainly not rigorous, especially in web crawlers, there is a risk of losing valuable data.

2 Common functions in tenacity

As a third-party Python library, we can install it using pip install tenacity. After the installation is complete, let's learn the main usage and features of tenacity:

2.1 basic use of tenacity

Tenacity's error retry core function is implemented by its retry decorator. When parameters are not passed to the retry decorator by default, it will keep retrying when the function it decorates throws an error, such as the following simple example:

Import randomfrom tenacity import retry@retrydef demo_func1 (): a = random.random () print (a) if a > = 0.1: raise Exceptiondemo_func1 ()

As you can see, every time our function generates a random number between 0 and 1, it will stop throwing an error when the random number does not exceed 0.1, otherwise it will be caught by tenacity every time the error throwing behavior will be caught and retried immediately.

2.2 set maximum number of retries

Sometimes there is a limit to our tolerance for a retry of a function logic error. For example, when we call a network interface, if we fail n times in a row, we may think that the task itself is flawed. It can't be normal one day by retrying.

At this point, we can use the stop_after_attempt function in tenacity to pass in as the stop parameter in retry (), thus adding an end point to our "endless" error retry process, where stop_after_attempt () accepts an integer input as the maximum number of retries:

From tenacity import retry, stop_after_attempt@retry (stop=stop_after_attempt (3)) def demo_func2 (): print ('function execution') raise Exception demo_func2 ()

As you can see, after limiting the maximum number of retries, our function formally throws the corresponding Exception error in the function to end the retry process after 3 retries and still throws an error after the fourth execution.

2.3 set the maximum timeout for retry

In addition to setting the number of maximum error retries as in the previous section, tenacity also provides us with the stop_after_delay () function to set the maximum time spent for the entire retry process, which will also end the retry process:

Import timefrom tenacity import retry, stop_after_delay# set the maximum retry timeout to 5 seconds @ retry (stop=stop_after_delay (5)) def demo_func3 (): time.sleep (1) print (f 'has passed {time.time ()-start_time} seconds') raise Exception# recording start time start_time = time.time () demo_func3 ()

2.4 Combinatorial retry stop condition

If our task needs to add the maximum number of retries and the maximum timeout at the same time, we only need to use the | operator to combine different restrictions in tenacity and pass in the stop parameter of retry (). For example, in the following example, we can end the retry when our function executes retries for more than 3 seconds or more than 5 times:

Import timeimport randomfrom tenacity import retry, stop_after_delay, stop_after_attempt@retry (stop= (stop_after_delay (3) | stop_after_attempt (5)) def demo_func4 (): time.sleep (random.random ()) print (f 'has passed {time.time ()-start_time} seconds') raise Exception# recording start time start_time = time.time () demo_func4 ()

As you can see, in the above demonstration, the limit of "maximum 5 retries" is reached first, thus ending the retry process.

2.5 set the time interval between adjacent retries

In some cases, we do not want to start the next retry immediately after each retry throws an error. For example, in order to better camouflage our program in the crawler task, tenacity provides a series of very useful functions, together with the wait parameter of retry (), to help us properly handle the time interval between adjacent retries. There are two more practical ways:

2.5.1 set a fixed interval

We can set a fixed number of seconds between adjacent retries by using wait_fixed () in tenacity, as in the following simple example:

Import timefrom tenacity import retry, wait_fixed, stop_after_attempt# set the retry wait interval to 1 second @ retry (wait=wait_fixed (1), stop=stop_after_attempt (3)) def demo_func5 (): print (f 'has passed {time.time ()-start_time} seconds') raise Exception # record start time start_time = time.time () demo_func5 ()

2.5.2 set random time interval

In addition to setting a fixed time interval, tenacity can also help us set a uniformly distributed random number for neighboring retries through wait_random (). You only need to set the range of uniform distribution:

Import timefrom tenacity import retry, wait_random, stop_after_attempt# set the retry waiting interval to a random number between 1 and 3 @ retry (wait=wait_random (min=1, max=3), stop=stop_after_attempt (5)) def demo_func6 (): print ({time.time ()-start_time} seconds') raise Exception# record start time start_time = time.time () demo_func6 ()

It can be observed that the waiting time after each retry is random.

2.6 Custom whether to trigger a retry

The default policy of retry () in tenacity is to retry when the function it decorates "throws any error", but in some cases we may need to catch / ignore a specific type of error, or catch the result of an exception calculation.

Related practical features are also built into tenacity:

2.6.1 catch or ignore specific error types

Using retry_if_exception_type () and retry_if_not_exception_type () in tenacity, together with the retry parameter of retry (), we can catch or ignore specific types of errors:

From tenacity import retry, retry_if_exception_type, retry_if_not_exception_type@retry (retry=retry_if_exception_type (FileExistsError)) def demo_func7 (): raise TimeoutError @ retry (retry=retry_if_not_exception_type (FileNotFoundError)) def demo_func8 (): raise FileNotFoundError

2.6.2 Custom function result condition judgment function

We can write an additional condition judgment function to cooperate with retry_if_result () in tenacity to customize the condition judgment of the returned result of the function, and the retry operation will be triggered only when True is returned:

Import randomfrom tenacity import retry, retry_if_result@retry (retry=retry_if_result (lambda x: X > = 0.1) def demo_func9 (): a = random.random () print (a) return a # record start time demo_func9 ()

2.7 Statistics on error retries of functions

For a function decorated by tenacity's retry (), we can print its retry.statistics property to view the statistical record of error retries. For example, here we print the statistical results of the example function demo_func9 () executed earlier:

Demo_func9.retry.statistics

After reading this, the article "how to solve the problem of the most powerful error retest library in Python" has been introduced. If you want to master the knowledge of this article, you still need to practice and use it to understand it. If you want to know more about related articles, welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.