Ways of scaling out with Rebus #3: RabbitMQ

Posted on by mookid8000 Leave a comment

With RabbitMQ, scaling out your Rebus-based workers is easy too – in fact, I’ve created a small RabbitMQ scaleout demo that shows how a producer can produce small nuggets of work, and any number of consumers can be started in order to distribute work evenly.

The principle is the same as when you’re using Azure Service Bus: Just start a number of competing consumers and you’re good to go.

An example on running the producer with two consumers can be seen below – here, I have published 100 jobs for the consumers to share.

Competing consumers with RabbitMQ

If you’re interested in running the demo yourself, you can check out the code from GitHub and follow the instructions in the accompanying readme.

Ways of scaling out with Rebus #1

Posted on by mookid8000 2 Comments
Introduction

When you’re working with messaging, and you’re in need of processing messages that take a fair amount of time to process, you’re probably in need of some kind of scaling-out strategy. An example that I’ve been working with lately, is image processing: By some periodic schedule, I would have to download and render a number of SVG templates and pictures, and that number would be thousands and thousands.

Since processing each image would have no effect on the processing of the next image, the processing of images is an obvious candidate for some kind of parallelisation, which just happens to be pretty easy when you’re initiating all work with messages.

Rudimentary scaling: Increase number of threads

One way of “scaling out” your work with Rebus is to increase the number of worker threads that the bus creates internally. If you check out the documentation about the Rebus configuration section, you can see that it’s simply a matter of doing something like this:

Increasing the number of worker threads provides a simple and easy way to parallelise work, as long as your server can handle it. Each CLR thread will have 1 MB of RAM reserved for its stack, and will most likely require additional memory to do whatever work it does, so you’ll probably have to perform a few measurements or trial runs in order to locate a sweet spot where memory consumption and CPU utilization are good.

If you’re in need of some serious processing power though, you’ll most likely hit the roof pretty quickly – but you’re in luck, because your messaging-based app lends itself well to being distributed to multiple machines, although there are a few things to consider depending on the type of transport you’re using.

In the next posts, I’ll go through examples on how you can distribute your work and scale out your application when you’re using Rebus together with Azure Service Bus, RabbitMQ, SQL Server, and finally with MSMQ. Happy scaling!

Rebus transport: MSMQ vs. RabbitMQ

Posted on by mookid8000 3 Comments

Now that Rebus officially supports RabbitMQ, I’ll just outline some of the reasons why you might want to choose one over the other… here goes:

Pros of RabbitMQ

The Rabbit is faster!

In my experience, RabbitMQ is often slightly faster than MSMQ in simple scenarios. I know that people pump high volumes of messages through Rabbit every day, and people might throw around numbers like “100000 msg/s”, or “500000 msg/s” and stuff like that. It might be true, but I promise you that these volumes can only be achieved in a few cases where e.g. a bit of delivery guarantee, message durability and/or atomicity is traded for speed.

RabbitMQ is easier to scale out, though – where MSMQ doesn’t handle competing consumers very well, it’s definitely the way to go with Rabbit if you have to process large message volumes concurrently.

Rabbit is (in some ways) easier to manage

RabbitMQ is pretty easy to manage because it’s a server that is installed somewhere, and then you point all the clients to the broker and they’re good. It’s easy to just run it on port 5672, and then that’s the only port that needs to be opened for access through firewalls, across VLANs, and whatnot. Most serious installations will require at least two Rabbit nodes though, so you might need to account for some configuration time though.

It also comes with a fairly useful web-based management tool that allows you to inspect all queues on the Rabbit server. This centralized way of managing things just makes it feel like you’re in control.

RabbitMQ can be configured to route messages in many ways, and Rebus can leverage the multicast features of Rabbit to function as a global subscription storage. This means that you can do this:

(notice the “ ManageSubscriptions” above?) which will make Rebus use Rabbit to do the hard work when doing pub/sub. I’ll get back to this in a future blog post, I promise 🙂

Pros of MSMQ

More reliable

MSMQ will probably be more reliable in most scenarios, because applications are always talking to the locally hosted MSMQ service. I.e. MSMQ always does local store-and-forward of messages, which means that you can always count on being able to deliver a message – and that’s actually something!

MSMQ is inherently more distributed – i.e. it’s much more tolerant to machines rebooting, switches failing, etc etc – if it can synchronize messages across the network, it will do so – if not, it will back off and wait until it can do so again.

Works well in homogenous environment

And then, if you’re so lucky that your network consists of one LAN with Windows machines, then everything will just work! And that includes security with user rights on queues, etc.

Easy to install

Oh, and then MSMQ frickin’ comes with Windows out of the box. That’s pretty cool 🙂

The future

I hope to see both the MSMQ transport and the RabbitMQ transport thrive and prosper. I hope to use both myself, and I’d like to help people use both as well.

The end

That was a long post! I hope even more people will find Rebus useful now that it can work with RabbitMQ. As always, just go get the bits from NuGet by doing the usual install-package rebus.rabbitmq.

New officially supported Rebus transport: RabbitMQ

Posted on by mookid8000 1 Comment

It’s been long underway, as I started developing it almost immediately after starting the Rebus project, so it’s almost as old as Rebus itself.

I originally started just out of curiosity, and because I wanted to learn about the AMQP model, but after having convinced a rapidly growing Aarhus trading firm to use Rebus for their enterprise messaging needs, and they had some bad experiences with MSMQ as their messaging infrastructure across VLANs, I was encouraged/forced to take it more seriously… and now… finally… I’m ready to announce that RabbitMQ is an official Rebus transport!

What does that mean?

It means that Rebus will exhibit the same guarantees with regards to atomiticy and transactionality of message exchange with RabbitMQ as it will with MSMQ! At first I didn’t think this was achievable, but it seems RabbitMQ has just enough transactional capability and Rebus requires sufficiently little of its queueing infrastructure, that everything just ends up working perfectly!

It also meant that we – the other day when I did a Rebus code camp with the local .NET user group, and some people had trouble making the MSMQ of their ridiculously locked-down work laptops talk to my MSMQ – could switch from MSMQ to Rabbit by going from

to

and everything would work, even for those who were struggling with their frigid laptops… pretty cool, huh?

There are still differences, however – in the next post I’ll go through some of the reasons why you might choose one over the other…