I deliberately waited until the last post in my series on how to scale your Rebus work to talk about how to do it with MSMQ. The thing is, MSMQ doesn’t lend itself well to the competing consumers pattern and thus requires some other mechanism in order to be able to effectively distribute work between a number of workers.
NServiceBus achieves this ability with its Distributor process, which is a sophisticated load balancer that communicates with its worker processes in order to avoid overloading them with work, etc.
The Rebus way of doing this is much simpler. I’ve long contemplated building a simple load balancer – one that blindly distributes work in a round-robin style fashion, which I believe would provide a sufficiently powerful and wonderfully unsophisticated way of scaling out your chunks of work.
This means that in a producer/consumer scenario, you would send all the work to the distributor, which would forward pieces of work to consumers in equal portions. It can be shown like this:
This is so simple, it cannot fail.
Which is why I’ve made the Rebus.MsmqLoadBalancer package, containing a LoadBalancerService.
Once you’ve executed the following Powershell spell:
|
1 |
install-package rebus.msmqloadbalancer -projectname YourProject |
the load balancer service can be started like this (assuming YourProject was a console application):
|
1 2 3 4 5 6 7 8 9 10 11 |
using(var loadBalancer = new LoadBalancerService("distributor")) { loadBalancer.AddDestinationQueue("worker@machine1"); loadBalancer.AddDestinationQueue("worker@machine2"); loadBalancer.AddDestinationQueue("worker@machine3"); loadBalancer.Start(); Console.WriteLine("Press ENTER to stop the load balancer"); Console.ReadLine(); } |
In this example, the load balancer will receive messages from the local distributor queue and forward them, distributing them evenly among the three workers residing on machine1, machine2, and machine3.
If you want to put this bad boy to some serious action, you’ll most likely want to wrap it in some kind of host process – either hosting it in the same process as the producer of your work, or in a Topshelf service or something.
One thing to note though: I haven’t had the chance to try the load balancer myself yet, mainly because my own horizontal scaling experience comes from working with RabbitMQ and Azure Service Bus only.
If you’re interested in using it, please don’t hesitate to contact me with questions etc., and I’ll be happy to help you on your way!