Ways of scaling out with Rebus #2: Azure Service Bus

Posted on by mookid8000 Leave a comment

Scaling out your application is easy with Azure Service Bus, because Azure Service Bus by design lends itself well to the competing consumers pattern as described by Gregor Hohpe and Bobby Woolf in the Enterprise Integration Patterns book.

So, in order to make this post a little longer, I will tell a little bit on how Rebus makes use of Azure Service Bus. And then I’ll tell you how to scale it 🙂

Rebus and queue transactions

When Rebus is configured to use Azure Service Bus to transport messages like this:

the bus will not use Azure Service Bus queues for its input queue and error queue, as you might think.

This is because Rebus will go to great lengths to promise you that a message can be received, and 0 to many messages sent – in one single queue transaction!

This means that the underlying transport layer must somehow be capable of receiving and sending messages atomically – and in a way that can be either committed or rolled back.

And since Azure Service Bus has limited transactional capabilities that do NOT allow for sending messages to multiple queues transactionally, we had to take a different approach with Rebus.

So, how does Rebus actually use Azure Service Bus?

What Azure Service Bus DOES support though, is receiving and sending atomically within one single topic.

So when Rebus starts up with Azure Service Bus, it will ensure that a topic exists with the name “Rebus”, which will be used to publish all messages that are sent.

And then, for each logical input queue – let’s call it “some_input_queue” – there will be a subscription for that queue by the same name, and that subscription will be configured with a SqlFilter that filters the received messages on a specific message property that holds the name of the intended recipient’s input queue. The filter will then ensure that only the intended messages are received for that endpoint.

So – how to scale it?

Easy peasy – in the Azure portal, go to this section of your cloud service:

Skærmbillede 2013-12-18 kl. 15.01.25

and go crazy with this bad boy:

Skærmbillede 2013-12-18 kl. 15.01.39

and – there you have it! – that is how you can scale out your work with Rebus in Azure 🙂

One thing, though – when you’re doing some serious number crunching, depending on the granularity of your messages of course, you may be bitten by the fact that Azure Service Bus’ BrokeredMessage’s lease expires after 60 seconds – if that is the case, Rebus has a fairly non-intrusive way of letting you renew the lease, which you can read more about in the “more about the Azure Service Bus transport” on the Rebus wiki.

In the next post, I’ll delve into how to scale your Rebus workers if you’re using RabbitMQ.

Ways of scaling out with Rebus #1

Posted on by mookid8000 2 Comments
Introduction

When you’re working with messaging, and you’re in need of processing messages that take a fair amount of time to process, you’re probably in need of some kind of scaling-out strategy. An example that I’ve been working with lately, is image processing: By some periodic schedule, I would have to download and render a number of SVG templates and pictures, and that number would be thousands and thousands.

Since processing each image would have no effect on the processing of the next image, the processing of images is an obvious candidate for some kind of parallelisation, which just happens to be pretty easy when you’re initiating all work with messages.

Rudimentary scaling: Increase number of threads

One way of “scaling out” your work with Rebus is to increase the number of worker threads that the bus creates internally. If you check out the documentation about the Rebus configuration section, you can see that it’s simply a matter of doing something like this:

Increasing the number of worker threads provides a simple and easy way to parallelise work, as long as your server can handle it. Each CLR thread will have 1 MB of RAM reserved for its stack, and will most likely require additional memory to do whatever work it does, so you’ll probably have to perform a few measurements or trial runs in order to locate a sweet spot where memory consumption and CPU utilization are good.

If you’re in need of some serious processing power though, you’ll most likely hit the roof pretty quickly – but you’re in luck, because your messaging-based app lends itself well to being distributed to multiple machines, although there are a few things to consider depending on the type of transport you’re using.

In the next posts, I’ll go through examples on how you can distribute your work and scale out your application when you’re using Rebus together with Azure Service Bus, RabbitMQ, SQL Server, and finally with MSMQ. Happy scaling!