Shulou(Shulou.com)06/02 Report--

This article will explain in detail how Python uses backoff to achieve polling more elegantly. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

We often encounter such a scenario in development, that is, round robin operation. Today, we introduce a Python library, which is used to achieve the effect of round robin more easily-backoff.

Introduction and installation of backoff module

This module mainly provides a decorator to decorate the function so that it will retry when certain conditions are encountered (that is, execute the decorated function repeatedly). It is usually suitable for us to obtain some unreliable resources, such as resources that will break down intermittently.

In addition, the decorator supports normal synchronous methods as well as asynchronous asyncio code.

The installation of the backoff module is also simple, which can be done through pip:

Pip install backoff

Backoff usage and simple source code analysis

Backoff provides two main decorators, through backoff. Call, we can see these two decorators by prompting us, they are:

Backoff.on_predicatebackoff.on_exception

Check the source code of backoff through github. The source code directory backoff/_decorator.py is defined as follows:

Def on_predicate (wait_gen, predicate=operator.not_, max_tries=None, max_time=None, jitter=full_jitter, on_success=None, on_backoff=None, on_giveup=None, logger='backoff', * * wait_gen_kwargs): # the specific code is omitted # the definition of each parameter is clearly explained in the source code passdef on_exception (wait_gen, exception, max_tries=None, max_time=None, jitter=full_jitter, giveup=lambda e: False, on_success=None, on_backoff=None, on_giveup=None, logger='backoff' * * wait_gen_kwargs): # omit the specific code # the definition of each parameter is clearly explained in the source code pass

As you can see, many parameters are defined, all of which are explained in detail in the source code. Here is a brief introduction:

First, wait_gen: indicates the length of time, in seconds, to wait for each loop. Its type is a generator, and three generators are built into backoff. Let's look at the source code, the directory is backoff/_wait_gen.py. Let's take a detailed implementation of one of them:

# omit the implementation code # base * factor * ndef expo (base=2, factor=1, max_value=None): "Generator for exponential decay. Args: base: The mathematical base of the exponentiation operation factor: Factor to multiply the exponentation by. Max_value: The maximum value to yield. Once the value in the true exponential sequence exceeds this, the value of max_value will forever after be yielded." N = 0 while True: a = factor * base * * n if max_value is None or a < max_value: yield an + = 1 else: yield max_value# controls def fibo (max_value=None): pass# constant value def constant (interval=1): pass through Fibonacci sequence

From the source code, it is not difficult to see that through some strategies, each yield returns a different value, which is the number of seconds to wait for the retry. Of course, because this parameter type is a generator, obviously we can also customize it. At the same time, we will find that each wait_gen is parameter-controlled, so we should be able to modify the initial value of this parameter.

Obviously, wait_gen_kwargs is used to pass these parameters. It is controlled by variable keyword parameters and can be passed directly in the form of key=value. A simple example is as follows:

@ backoff.on_predicate (backoff.constant, interval=5) def main3 (): print ("time is {} retry..." .format (time.time ()

Predict and exception. These two are relatively simple. Predict accepts a function that retries when the function returns True, otherwise it stops, and the function accepts an argument whose value is the return value of the decorated function. The default value for this parameter is: operator._not. The source code for this function is as follows:

Def not_ (a): "Same as not a." Return not a

So the default return is the return value of the not decorated function. If True is returned when the decorated function does not return a value, a retry will be made.

The sample code is as follows:

Import backoffimport time@backoff.on_predicate (backoff.fibo) def test2 (): print ("time is {}, retry..." .format (time.time () if _ _ name__ = = "_ _ main__": test2 () # is equivalent to: # must accept a parameter The value of this parameter is the return value of the decorated function def condition (r): return True @ backoff.on_predicate (backoff.fibo, condition) def test2 (): print ("time is {}, retry..." .format (time.time () if _ _ name__ = "_ _ main__": test2 ()

The implementation results are as follows:

$python3 backoff_test.pytime is 1571801845.834578, retry...time is 1571801846.121314, retry...time is 1571801846.229812, retry...time is 1571801846.533237, retry...time is 1571801849.460303, retry...time is 1571801850.8974788, retry...time is 1571801856.498335, retry...time is 1571801861.56931, retry...time is 1571801872.701226, retry...time is 1571801879.198495, retry.

There are several points to pay attention to:

If you customize the function corresponding to this parameter, the function needs to accept a parameter whose value is the return value of the decorated function. We can make some conditional judgments by controlling the return value, and the retry ends when some special conditions are reached.

In the example, wait_gen uses backoff.fibo. Pay attention to the time interval of the output. The time interval here does not seem to be the same as the number of intervals returned by fibo. In fact, to achieve this effect, we need to set the jitter parameter to None, which will be explained later when we introduce the jitter parameter.

Exception is an instance that accepts an exception type, which can be a single exception or multiple exceptions in the form of tuples. A simple example is as follows:

Import timeimport randomimport backofffrom collections import dequeclass MyException (Exception): def _ init__ (self, message, status): super (). _ _ init__ (message, status) self.message = message self.status = statusclass MyException2 (Exception): pass@backoff.on_exception (backoff.expo, (MyException, MyException2) def main (): random_num = random.randint (0,9) print ("retry...and random num is {}" .format (random_num) if random_num% 2 = 0: raise MyException ("my exception") Int ("1000" + str (random_num)) raise MyException2 ()

Max_tries and max_time are also relatively simple, representing the maximum number of retries and the longest retry time, respectively. There will be no demonstration here.

Giveup in @ backoff.on_exception, which accepts an exception instance and determines whether the loop needs to continue by making some conditional judgments on the instance. If True is returned, it ends and vice versa. The default value is always to return False, that is, to loop all the time. Examples are as follows:

Import randomimport backoffclass MyException (Exception): def _ init__ (self, message, status): super (). _ _ init__ (message, status) self.message = message self.status = statusdef exception_status (e): print ('exception status code is {}' .format (e.status)) return e.status% 2 = 0 @ backoff.on_exception (backoff.expo, MyException, giveup=exception_status) def main (): random_num = random.randint (0 9) print ("retry...and random num is {}" .format (random_num)) raise MyException ("my exception", int ("1000" + str (random_num)) if _ _ name__ = = "_ _ main__": main ()

Running result:

Retry...and random num is 5exception status code is 10005retry...and random num is 0exception status code is 1000 goes through the raise code again. So the exception will still be thrown Traceback (most recent call last): File "backoff_test.py", line 36, in main () File "/ Users/ruoru/code/python/exercise/.venv/lib/python3.7/site-packages/backoff/_sync.py", line 94, in retry ret = target (* args, * * kwargs) File "backoff_test.py", line 32, in main raise MyException ("my exception") Int ("1000" + str (random_num)) _ _ main__.MyException: ('my exception', 10000)

There are two points to note:

The function accepted by this parameter still has only one parameter, and the value of this parameter is an exception instance object.

As we can see from the result, when an exception is thrown, it will first enter the function accepted by giveup, and if the function determines that giveup is needed, the current exception will still be thrown. So if necessary, the code still needs to do exception logic handling.

On_success, on_backoff and on_giveup are the same class of parameters for event handling:

The on_sucess event is a bit more difficult to understand, which means that the decorated function will exit when the round robin ends successfully, and for on_exception, on_success will be called when no exception occurs. In the case of on_predicate, it is called if the loop is returned to False by the predicate keyword.

On_backoff is called when the program generates a loop

On_giveup is called when the program reaches the current maximum number of attempts. For on_predicate, it is called if it is through max_tries or max_time, and for on_exception, on_giveup is also called when True is returned for the exception parameter.

To sum up, the direct control end of max_tries and max_time calls on_giveup, and the exception parameter also ends the program by returning True, which is used to control the end of the program, so on_giveup will also be called. While the predicate parameter returns True, the program continues, which is used to control whether the program continues to wander, so when it ends, it calls on_success.

The experimental code is as follows:

'' @ Author: ruoru@Date: 2019-10-22 15:30:32@LastEditors: ruoru@LastEditTime: 2019-10-23 14:37:13@Description: backoff'''import timeimport randomimport backoffclass MyException (Exception): def _ init__ (self, status, message): super (). _ init__ (status Message) self.status = status self.message = messagedef backoff_hdlr (details): print ("Backing off {wait:0.1f} seconds afters {tries} tries"calling function {target} with args {args} and kwargs"{kwargs}" .format (* * details) def success_hdlr (details): print ("Success offafters {tries} tries"calling function {target} with args {args} and kwargs"{kwargs}" .format (* * details)) def giveup_hdlr (details ): print ("Giveup off {tries} tries"calling function {target} with args {args} and kwargs"{kwargs}" .format (* * details)) @ backoff.on_predicate (backoff.constant # continue when random num is not equal to 10009 # when random_num equals 10009 On_success lambda x: X! = 10009, on_success=success_hdlr, on_backoff=backoff_hdlr, on_giveup=giveup_hdlr, max_time=2) def main (): num = random.randint (10000, 10010) print ("time is {}, num is {}, retry..." .format (time.time (), num)) return num@backoff.on_exception (backoff.constant, MyException, # exit when the status of the Exception instance object is 10009 # when the condition is true Call on_giveup giveup=lambda e: e.status = = 10009, on_success=success_hdlr, on_backoff=backoff_hdlr, on_giveup=giveup_hdlr,) def main2 (): num = random.randint (10000, 10010) print ("time is {}, num is {}, retry..." .format (time.time (), num)) # if it is established by this condition On_success if num = = 10010: return raise MyException (num, "hhh") if _ _ name__ = = "_ _ main__": # main () main2 ()

The logger parameter, which is obviously used to control log output, will not be described in detail here. An example of copy official documentation:

My_logger = logging.getLogger ('my_logger') my_handler = logging.StreamHandler () my_logger.add_handler (my_handler) my_logger.setLevel (logging.ERROR) @ backoff.on_exception (backoff.expo, requests.exception.RequestException, logger=my_logger) #...

The last parameter, jitter, does not quite understand the function of this parameter at first, as explained in the document:

Jitter: A function of the value yielded by wait_gen returning the actual time to wait. This distributes wait times stochastically in order to avoid timing collisions across concurrent clients. Wait times are jittered by default using the full_jitter function. Jittering may be disabled altogether by passing jitter=None.

A little dizzy, so I went to see the source code and understood the usage. The key source code was intercepted as follows:

# backoff/_decorator.pydef on_predicate (wait_gen, predicate=operator.not_, max_tries=None, max_time=None, jitter=full_jitter, on_success=None, on_backoff=None, on_giveup=None, logger='backoff', * * wait_gen_kwargs): pass # omitted # because asynchrony is not used So if retry is None: retry = _ sync.retry_predicate# backoff/_sync# analysis shows that there is a sentence to get the next wait duration seconds = _ next_wait (wait, jitter, elapsed, max_time_) # backoff/_commondef _ next_wait (wait, jitter, elapsed, max_time): value = next (wait) try: if jitter is not None: seconds = jitter (value) else: seconds = value except TypeError: warnings.warn ("Nullary jitter function signature is deprecated. Use "" unary signature accepting a wait value in seconds and "" returning a jittered version of it. ", DeprecationWarning, stacklevel=2,) seconds = value + jitter () # don't sleep longer than remaining alloted max_time if max_time is not None: seconds = min (seconds, max_time-elapsed) return seconds

If you look at the first few lines of code, it should be clear that if jitter is None, the value value returned by the first parameter will be used, and if used, the algorithm will be done on this value value again, default to full_jitter (value). Backoff/_jitter.py provides two algorithms, the code is not long, post it to have a look:

Import randomdef random_jitter (value): "" Jitter the value a random number of milliseconds. This adds up to 1 second of additional time to the original value. Prior to backoff version 1.2 this was the default jitter behavior. Args: value: The unadulterated backoff value. "" Return value + random.random () def full_jitter (value): "Jitter the value across the full range (0 to value). This corresponds to the" Full Jitter "algorithm specified in the AWS blog's post on the performance of various jitter algorithms. (http://www.awsarchitectureblog.com/2015/03/backoff.html) Args: value: The unadulterated backoff value." Return random.uniform (0, value) about how Python uses backoff to achieve polling more elegantly is shared here. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.