Awesome Python Library: Tenacity

Link: https://tenacity.readthedocs.io/en/latest/

When writing code or tests in Python, one issue I had was when the code would fail due to random things like network issues or external peripherals not responding in time. Just rerunning the tests would make them pass. The unreliability wasn't in the code but in things out of my control, like network issues.

So I had to add extra code to retry the code, but this added unnecessary complexity.

Thats when I discovered the Tenacity library and it saved me hours and a lot of useless boilerplate code.

Some unreliable code

Let's try some dummy unreliable code.

import time
import random


FAIL_PERCENT= 0.7 # 70% of time

def unreliable_function():
    print("In unreliable function:")
    if random.random() < FAIL_PERCENT:
        raise Exception("Operation failed randomly")
    else:
        # Successful operation
        return "Success"

The code uses the random function to throw an exception 70% of the time. Try running it, it will fail more often than not.

Normally, I would add some code to retry the code on failure:

for i in range(10):
    try:
        unreliable_function()
        print("passed, yay!")
        break
    except Exception as e:
        print("Function returned error, sleeping")
        time.sleep(1)

Running it, I get output like:

In unreliable function:
Function returned error, sleeping
In unreliable function:
Function returned error, sleeping
In unreliable function:
Function returned error, sleeping
In unreliable function:
passed, yay!

So the code works, but I have to manually add try excepts and a sleep. And then I have to maintain the code.

Tenacity: Retry on failure

Let's see how we can retry the buggy code with Tenacity:

@retry    # <-- This is the only NEW code
def unreliable_function():
    print("In unreliable function:")
    if random.random() < FAIL_PERCENT:
        raise Exception("Operation failed randomly")
    else:
        # Successful operation
        return "Success"

unreliable_function()

The simplest thing is to just add the @retry decorator to the code. It will keep running the code if it passes.

The fact you have no extra code, just a decorator, means the code is really easy to follow when someone else picks up your code a few months/years from now.

But what if we want to only try X times, and sleep in between tries?

@retry(wait = wait_fixed(1), stop = stop_after_attempt(8))
def unreliable_function():
    print("In unreliable function:")
    if random.random() < FAIL_PERCENT:
        raise Exception("Operation failed randomly")
    else:
        # Successful operation
        return "Success"

unreliable_function()

In unreliable function:
In unreliable function:
In unreliable function:

'Success'

The wait = wait_fixed(1) will wait 1 second between runs. Instead of wait_fixed we can also wait a random time, or wait_exponential() which will increase the time exponentially.

stop = stop_after_attempt(8) will stop after 8 tries. You can also set a maximum timeout– like the code must fail after 30 seconds.

Adding multiple conditions

We can also chain conditions. So for example, you want to wait between retries, but you don't want to spend hours on it as you have other things to do (or you might have other tests to run, and want to fail early).

@retry(stop=(stop_after_attempt(10)  | stop_after_delay(30) ) )

The above will retry the code 10 times, but for a maximum of 30 seconds. If after 30 seconds, the code is still failing, it will return with the error message.

Another cool function if Custom Callbacks, where you can have your own test to check if the code failed.

Say you are reading a webpage and it returns a 500 server error HTTP code. While there has been no exception, the code still failed.

In this case, you can write your own custom code to check the HTTP code is 200.

In conclusion

I wasted a lot of time writing my own code to retry functions in various cases, and it became complex and unmanageable soon. Tenacity is so simple to use, I recommend just using it.