Retry flaky task in Python using Tenacity

Photo by Tai Bui on Unsplash

Flaky events needs retry action to achieve desire results. In python Tenacity module gives a mechanism to handle flaky events with retry decorator.

Tenacity is a task retrying library written in Python. The simplest use case for Tenacity is to retry a flaky function whenever an exception occurs until a successful value is returned.

Installation

$ pip install tenacity

Let’s look at a simple use case for the Tenacity module. In the following code, the defined task run_my_task is intentionally made flaky, which means that there is no guarantee of success when the task is executed. The @retry decorator re-executes the task until it returns a success, with no limit on the number of retry attempts.

import random
from tenacity import retry

@retry
def run_my_task():
  """ Intentionally making this task to fail randomly. """
  if random.randint(0, 10) > 1:
    raise IOError("My Task Failed With IOError.")
  else:
    print("My Task Works!!!")

print(run_my_task())

There are several conditions that can be attached to the retry logic, such as stop conditions, wait conditions, customizing retrying based on exceptions and expected return values, retrying on coroutines, and retrying code blocks with context managers. Here are some examples of these conditions.

Stop Conditions

Stop retrying after a certain number of attempts. This condition can be added to the @retry decorator using stop=stop_after_attempt(<number of attempts>). In the following example, retrying will happen until 5 attempts have been made, at which point the code will raise an exception.

from tenacity import retry, stop_after_attempt

@retry(stop=stop_after_attempt(5))
def my_task():
    print("I will stop after 5attempt.")
    raise Exception

Stop retrying after a certain delay. The @retry decorator's stop argument can be set to stop_after_delay(<delay in seconds>), which will cause retrying to continue until the specified delay has passed.

from tenacity import retry, stop_after_delay

retry(stop=stop_after_delay(10))
def run_my_task():
    print("Stopping after 10 seconds")
    raise Exception

This will cause run_my_task to be retried until 30 seconds have passed since the first attempt. Note that this does not necessarily mean that retrying will stop after 30 seconds; it just means that retrying will continue until the 30-second delay has elapsed. The actual time at which retrying stops will depend on other factors, such as the wait condition.

The stop condition in the @retry decorator can be a combination of multiple conditions using the | (logical OR) operator. For example:

from tenacity import retry, stop_after_delay, stop_after_attempt

retry(stop=(stop_after_delay(30) | stop_after_attempt(5)))
def my_task():
    print("Stopping after 10 seconds or 5 attempts.")
    raise Exception

This will cause run_my_task to be retried until either 5 attempts have been made or 30 seconds have passed since the first attempt, whichever occurs first. Note that the stop condition is evaluated after each attempt, so if the number of attempts reaches 5 before the 30-second delay has elapsed, retrying will stop even if the delay has not yet passed.

Waiting before retrying

Between two retry event certain conditions can be attached like wait_fixed , wait_random, wait_exponential, etc. and these are the values are getting pass to the “wait” argument.

This is an example of keeping wait time 4 seconds between the retry events.

from tenacity import retry, wait_fixed

@retry(wait=wait_fixed(4))
def my_task():
    print("Wait 4 seconds between retries")
    raise Exception

OR it can be a random wait time between specified min and max limits in seconds.

from tenacity import retry, wait_random

@retry(wait=wait_random(min=10, max=20))
def my_task():
    print("Wait randomly  10 to 20 seconds between retries.")
    raise Exception

This way there are few more conditions we can specify as mentioned below.

1. Wait exponentially, In this wait time is increasing exponentially after each retry. This usually helps distributed system environments. Exponential condition takes these three arguments multiplier, min and max.
2. wait random exponential, This is same as above, only difference is multiplier value is getting randomly picked.
3. Wait chain, Using this we can make conditional wait time e.g. wait 3 seconds for 3 attempts, wait 10 seconds for 5 atempts and the default wait time for retry after that.

from tenacity import retry, wait_chain, wait_fixed

@retry(wait=wait_chain(*[wait_fixed(3) for i in range(3)] +
                       [wait_fixed(10) for i in range(5)] +
                       [wait_fixed(9)]))
def my_task():
    print("Wait 3s for 3 attempts, 7s for the next 2 attempts and 9s for all attempts thereafter")
    raise Exception

Conditional retry on exception type.

Retry condition can be set upon the exception raised by the code. e.g. Retry whenever IOError exception is raised or retry if other than IOError exception raised.

from tenacity import retry, retry_if_exception_type, retry_if_not_exception_type

@retry(retry=retry_if_exception_type(IOError))
def my_task():
    print("Retry forever with no wait if an IOError occurs, raise any other errors")
    raise Exception

@retry(retry=retry_if_not_exception_type(IOError))
def my_task2():
    print("Retry forever with no wait if any error other than IOError occurs. Immediately raise IOError.")
    raise Exception

Provide a custom function returns to alter retry behaviour.

from tenacity import retry, retry_if_result

def is_none_p(value):
    """Return True if value is None"""
    return value is None

@retry(retry=retry_if_result(is_none_p))
def might_return_none():
    print("Retry with no wait if return value is None")

Other Features

1. Custom Callbacks. We can define a custom callback functions when all retries failed. Retry accept callback function to this “retry_error_callback” argument.
2. Retry can be used for Async function.
3. We can execute action after and before of calling the function by using “After” and “before” argument passing to the retry call.

This is short notes about tenacity module and as per my experience it is very convenient to use while connecting to sockets, waiting for resource to be ready etc.