Azure Queue Agent Library

Azure Queue Agent (short: AQuA) is an open-source lightweight library to help you build Azure Queue based producer/consumer scenarios to handle execution of jobs, tasks, or whatever else you may want to call it. The complete sources are available on GitHub. The library is also available as a NuGet package for your convenience.

How does Azure Queue Agent work?

AQuA comes with two main components: a Producer and a Consumer. As the names suggest, the Producer can be used to produce (i.e. enqueue) new jobs, while the Consumer can be used to consume (i.e. dequeue and then handle) jobs from the queue. Job Descriptors are used to define which job should be executed and what parameters should be used for the execution. They are encoded as simple JSON objects like the one below, i.e. they can also easily be written manually which makes also them perfect candidates for use with the Azure Scheduler and help you run your tasks on recurring schedules. That said, with AQuA it is very simple to build a scalable and robust collection of workers which take care of all your background processing jobs.

{ "Job": "HelloWho", "Properties": { "Who": "World" } }

The above example for instance would queue the HelloWho job, which does nothing more but print the value of the Who parameter on stdout like this: “Hello, <Who>!”. In addition, the Azure Queue Agent Consumer can be configured to either delete or requeue messages which were badly formatted, using unknown jobs or which could not be executed successfully, such that you can even use a single queue for multiple different pools of workers, should you ever find yourself in that situation.
How to use Azure Queue Agent Consumers
I assume you already have an Azure account, but if not, you can start with the free trial. If you just want to test it locally, you don’t even need an Azure account, the Azure Storage Emulator will do. First, get the NuGet package:
Install-Package aqua.lib

or for pre-release packages:
Install-Package aqua.lib -Pre

Second, you’ll need to tell the producer and/or consumer where to enqueue/dequeue messages, i.e. which queue in which storage account. You also need to tell it just a little bit more about the jobs you can process, by providing a JobFactory in which you register all your known job types. In the future I will likely add a JobFactory which searches for all classes which implement IJob in the current Process/AppDomain so you won’t need to manually register jobs; however, the manual registration certainly gives you much more control. For the default storage account from the emulator, you just need to provide the name of the queue (which by the way is subject to some restrictions by Azure). The queue does not have to exist, since the producer/consumer will create it for you if it doesn’t.
// Setup and initialization
JobFactory factory = new JobFactory();

// Register all the jobs you want your consumer to be able to handle.
factory.RegisterJobType(typeof(HelloWho));
factory.RegisterJobType(typeof(MyBackgroundJob));

// Use the storage account from the emulator with queue "jobs".
ConnectionSettings connection = new ConnectionSettings("jobs");

Producer producer = new Producer(connection, factory);
Consumer consumer = new Consumer(connection, factory);

Producing messages programmatically is then as simple as the following:
HelloWho job = new HelloWho() { Who = "Azure Queue Agent Example" };
producer.One(job);

And consuming one single message can be done as follows.
consumer.One();

Now of course, if the queue is empty (i.e. no jobs are pending), then there is nothing to consume, and the above example would return immediately. If however you want to wait a bit and retry for a while until a message is found, you can do that too. There are a couple of retry strategies that ship with the Azure Queue Agent library:

NoRetryStrategy — try only once and then give up; this is the default for the parameterless Consumer.One() method.
SimpleRetryStrategy — try N times with a static wait time T between tries; both N and T are configurable.
LinearBackoffRetryStrategy — try N times with linearly increasing wait time T between tries; both N and T are configurable. E.g. wait 10 seconds, then 20 seconds, then 30 seconds, etc.
ExponentialBackoffRetryStrategy — try N times with exponentially increasing wait time T between tries; both N and T are configurable. E.g. wait 1 second, then 2 seconds, then 4 seconds, then 8 seconds, etc.
AddJitterRetryStrategy — add some randomization to the wait times of any other retry strategy. The randomization range can be configured in percentages, e.g. randomize in the range of 75% to 125%.

For example, if you want to use an exponential back-off strategy with some randomization added:
IRetryStrategy dequeueStrategy = new AddJitterRetryStrategy(
    // Retry at most 9 times (so at most 10 tries in total)
    new ExponentialBackoffRetryStrategy(9, TimeSpan.FromSeconds(1)),
    0.5,  // Randomization Range Minimum:  50%
    1.2); // Randomization Range Maximum: 120%
consumer.one(dequeueStrategy);

You can also build a retry strategy that is tailored to your very own needs, simply by implementing the IRetryStrategy interface. Back-off strategies can be built by deriving the BackoffRetryStrategyBase class. Usually, for workers you will want to run Consumer.One() in a loop so it will continuously process jobs.
More Information
You can find more information in my blog. Posts are listed from oldest to newest to help you read in chronological order.

Introduction