Now more than ever I’m finding that the systems I write are required to communicate with a whole host of other systems. With “cloud computing” becoming ever more popular this trend will no doubt continue. Two obvious examples are databases and web-services (either directly or through APIs). Systems that rely on other applications and end-points naturally have extra failure points that we need to consider. More often than not, when calls to external end-points fail the problems are transient. That is to say if we try the same operation a little later the call magically works. There are many reasons why we experience transient errors, with some examples being insufficient band-width, transmission errors or servers being too busy to respond in a timely fashion. Of course, the end-point could be permanently down so our code should handle this gracefully, but usually pausing and retrying usually solves the problem.
Implementing a retry loop isn’t a difficult task, but given a class that calls a number of different methods of the same remote end-point, we don’t want to duplicate the retry code for each call. Fortunately, a relatively new feature in .NET makes implementing generic retry loops much easier. Using our old friend the console application (and with the same disclaimer about client profiles as always) I’ve written a calculator class that we will be calling using a retry loop:
using System;
namespace RetryLoops
{
public class Calculator
{
private readonly Random random;
public Calculator()
{
unchecked
{
random = new Random((int)DateTime.Now.Ticks);
}
}
public int Add(int x, int y)
{
var value = random.Next(0, 5);
if (value == 0)
{
throw new InvalidOperationException("Transient error!");
}
return x + y;
}
public int Subtract(int x, int y)
{
var value = random.Next(0, 5);
if (value == 0)
{
throw new InvalidOperationException("Transient error!");
}
return x - y;
}
}
}
Note that the Add and Subtract methods have a 1 in 5 chance of throwing an exception when they’re called. This is to simulate transient errors. Note also that I’m using the seeding technique for random numbers that I introduced in my last post.
I’ve coded the Program class as follows:
using System;
namespace RetryLoops
{
public static class Program
{
public static void Main()
{
var calculator = new Calculator();
Console.WriteLine(calculator.Add(1, 2));
Console.WriteLine(calculator.Subtract(1, 2));
Console.WriteLine(calculator.Add(1, 2));
Console.WriteLine(calculator.Subtract(1, 2));
Console.WriteLine(calculator.Add(1, 2));
Console.WriteLine(calculator.Subtract(1, 2));
Console.WriteLine(calculator.Add(1, 2));
Console.WriteLine(calculator.Subtract(1, 2));
Console.WriteLine(calculator.Add(1, 2));
Console.WriteLine(calculator.Subtract(1, 2));
Console.WriteLine(calculator.Add(1, 2));
Console.WriteLine(calculator.Subtract(1, 2));
Console.ReadLine();
}
}
}
You don’t need to run this very many times before you get an unhandled exception.
Adding a method called ExecuteOperationWithRetry to the program and changing the main method to use this gives:
using System;
using System.Threading;
namespace RetryLoops
{
public static class Program
{
public static void Main()
{
var calculator = new Calculator();
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Add(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Subtract(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Add(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Subtract(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Add(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Subtract(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Add(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Subtract(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Add(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Subtract(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Add(1, 2)));
Console.WriteLine(ExecuteOperationWithRetry(() => calculator.Subtract(1, 2)));
Console.ReadLine();
}
private static TReturn ExecuteOperationWithRetry<TReturn>(Func<TReturn> operation)
{
const int MaximumAttempts = 10;
var timeBetweenRetries = TimeSpan.FromMilliseconds(100);
var attempts = 0;
while (true)
{
try
{
attempts++;
return operation.Invoke();
}
catch (Exception exception)
{
// If the exception isn't transient then re-throw:
if (!(exception is InvalidOperationException))
{
throw;
}
// If we've had all our attempts already then re-throw:
if (attempts >= MaximumAttempts)
{
throw;
}
// Wait before making another attempt:
Console.WriteLine("Transient error encountered, retrying...");
Thread.Sleep((int)(timeBetweenRetries.TotalMilliseconds * Math.Pow(2, attempts - 1)));
}
}
}
}
}
As the operation parameter of ExecuteOperationWithRetry is of type Func
When I ran this I got the following:
…which proves the retry loop is doing its job.
To make the code even more generic we can add the following overload of the ExecuteOperationWithRetry method, enabling us to handle operations that don’t have return values:
private static void ExecuteOperationWithRetry(Action operation)
{
Func<bool> operationWithReturn = () =>
{
operation.Invoke();
return true;
};
ExecuteOperationWithRetry(operationWithReturn);
}
There is a lot of scope to improve this technique, and to tailor it to specific applications. In fact, it is possible to make a totally generic retry loop component that has the exceptions to retry, time to pause and number of attempts to make passed in as constructor arguments. I’ll cover this in a future article, but for now, this concludes my exploration of retry loops.