Thoughts on how to integrate with 3rd party libraries when you’re a library author

Yoda in a Santa's costume, carrying a magic wandWhen building .NET systems for clients, it’s just awesome how much free open source that’s available at your fingertips – and with tools like NuGet around, it’s become extremely easy to rapidly pull in various colors and sizes of Legos to snap together and build on. Except, sometimes those Legos don’t snap 🙁

Different types of libraries and frameworks are more or less susceptible to Lego-snap-issues, depending on their place in the stack. That is, libraries with many incoming references are inherently problematic in this regard, the ubiquitous examples being logging libraries and serialization libraries.

When I built Rebus, one of my solemn goals was to make Rebus dependent on .NET BCL ONLY. All kinds of integration with 3rd party libraries would have to be made via small dedicated adapter projects, because this way – in the case where there’s a conflict with one of your own dependencies – only that single adapter would have to be either built against your version of the problematatic dependency, or switched for something else.

Serialization with Rebus

Rebus serializes message bodies by using an ISerializeMessages abstraction. And since Rebus’ default transport message serialization format is JSON, and I didn’t feel like re-re-re-inventing any wheels, I decided to pull in the nifty JSON.NET package in order to implement Rebus’ JsonMessageSerializer that is part of the core Rebus.dll. But that clearly violates Rebus’ goal of having no dependencies besides what’s in the .NET framework – so how did I solve that?

Simple: ILmerge JSON.NET into Rebus! Simple solution, very effective.

Maybe there’s something I’m being ignorant about here, but I don’t get why projects like e.g. RavenDB keeps having issues with their JSON.NET dependency. Why didn’t Ayende just merge it from the start?

Update January 7th 2013: Well, it appears that they did just that for RavenDB 2: RavenDB-220

Logging with Rebus

And then there’s logging – in order to log things good, all Rebus classes must be given access to a logger – and I like it when the logger is named after the classes it is being used from, which always turns out to be a piece of really useful information when you’re trying to track down where stuff went wrong some time in the past.

Ideally, I wanted a syntax that resembles the Log4Net one-liner, that you’ll most likely encounter if you come across code that uses Log4Net for logging – each class usually initializes its own logger like this:

thus retrieving a static readonly logger instance named after the calling class. So, originally I started out by copying the Log4Net API, providing adapters for console logging, Log4Net logging, etc.

That worked perfectly for a while, but later on, since I spent so much time writing integration tests for Rebus, I also wanted the logging mechanism to be swappable at runtime – i.e. I wanted it so that Rebus could have its logger swapped from e.g. a ConsoleLogger to a NullLogger in a performance test, where all the console output would otherwise interfere with the measurements.

Therefore, my logging “abstraction” is a bit more involved. But it accomplishes all goals: All classes can have a static logger instance, named after the class, swappable at runtime, without introducting dependencies on external stuff, and with very little (read: an acceptable amount of) ceremony (too much for my tastes actually, but I accept this one because of the goodness it brings).

Check out what a class must do in order to get a Rebus logger:

As you can see, the logger is set when the Changed event is raised – and by taking control of the add/ remove operations of the event, I can ensure that the logger is set when each subscriber first subscribes. This way, the logger is immediately initialized by the currently configured static IRebusLoggingFactory, and in the event that the factory is changed, all logger instances are changed.

The Changed event is implemented like this:

The last thing is to provide a meaningful name to the logger – i.e. I wanted the name of the calling class to be used, so the call to GetCurrentClassLogger() had to do some StackFrame trickery in order to fullfil the promise made by its title. The implementation is pretty simple:

As you can see, it just skips one level on the call stack, i.e. it gets the calling method, and then we can use the standard reflection API to get the type that has this method. Easy peasy!

This was an example on how two of the more troublesome dependencies can be dealt with, allowing your library or framework to be effectively dependency-free.

I’ll be speaking at Warm Crocodile Developer Conference

In Copenhagen, on the 16th and 17th of January, there’s a new conference called Warm Crocodile – pretty cool title, if you ask me!

I begged and begged, until they caved in and gave me an hour of their otherwise extremely cool schedule – so I’ll be giving a talk that I’ve dubbed “Taking the hippie bus to the enterprise”… it will most likely be about using a free .NET service bus in combination with other cool free software to rapidly solve enterprisey problems without feeling a lot of pain.

If you’re interested, you can read the full abstract here: Taking the hippie bus to the enterprise

Rebus transport: MSMQ vs. RabbitMQ

Now that Rebus officially supports RabbitMQ, I’ll just outline some of the reasons why you might want to choose one over the other… here goes:

Pros of RabbitMQ

The Rabbit is faster!

In my experience, RabbitMQ is often slightly faster than MSMQ in simple scenarios. I know that people pump high volumes of messages through Rabbit every day, and people might throw around numbers like “100000 msg/s”, or “500000 msg/s” and stuff like that. It might be true, but I promise you that these volumes can only be achieved in a few cases where e.g. a bit of delivery guarantee, message durability and/or atomicity is traded for speed.

RabbitMQ is easier to scale out, though – where MSMQ doesn’t handle competing consumers very well, it’s definitely the way to go with Rabbit if you have to process large message volumes concurrently.

Rabbit is (in some ways) easier to manage

RabbitMQ is pretty easy to manage because it’s a server that is installed somewhere, and then you point all the clients to the broker and they’re good. It’s easy to just run it on port 5672, and then that’s the only port that needs to be opened for access through firewalls, across VLANs, and whatnot. Most serious installations will require at least two Rabbit nodes though, so you might need to account for some configuration time though.

It also comes with a fairly useful web-based management tool that allows you to inspect all queues on the Rabbit server. This centralized way of managing things just makes it feel like you’re in control.

RabbitMQ can be configured to route messages in many ways, and Rebus can leverage the multicast features of Rabbit to function as a global subscription storage. This means that you can do this:

(notice the “ ManageSubscriptions” above?) which will make Rebus use Rabbit to do the hard work when doing pub/sub. I’ll get back to this in a future blog post, I promise 🙂

Pros of MSMQ

More reliable

MSMQ will probably be more reliable in most scenarios, because applications are always talking to the locally hosted MSMQ service. I.e. MSMQ always does local store-and-forward of messages, which means that you can always count on being able to deliver a message – and that’s actually something!

MSMQ is inherently more distributed – i.e. it’s much more tolerant to machines rebooting, switches failing, etc etc – if it can synchronize messages across the network, it will do so – if not, it will back off and wait until it can do so again.

Works well in homogenous environment

And then, if you’re so lucky that your network consists of one LAN with Windows machines, then everything will just work! And that includes security with user rights on queues, etc.

Easy to install

Oh, and then MSMQ frickin’ comes with Windows out of the box. That’s pretty cool 🙂

The future

I hope to see both the MSMQ transport and the RabbitMQ transport thrive and prosper. I hope to use both myself, and I’d like to help people use both as well.

The end

That was a long post! I hope even more people will find Rebus useful now that it can work with RabbitMQ. As always, just go get the bits from NuGet by doing the usual install-package rebus.rabbitmq.

New officially supported Rebus transport: RabbitMQ

It’s been long underway, as I started developing it almost immediately after starting the Rebus project, so it’s almost as old as Rebus itself.

I originally started just out of curiosity, and because I wanted to learn about the AMQP model, but after having convinced a rapidly growing Aarhus trading firm to use Rebus for their enterprise messaging needs, and they had some bad experiences with MSMQ as their messaging infrastructure across VLANs, I was encouraged/forced to take it more seriously… and now… finally… I’m ready to announce that RabbitMQ is an official Rebus transport!

What does that mean?

It means that Rebus will exhibit the same guarantees with regards to atomiticy and transactionality of message exchange with RabbitMQ as it will with MSMQ! At first I didn’t think this was achievable, but it seems RabbitMQ has just enough transactional capability and Rebus requires sufficiently little of its queueing infrastructure, that everything just ends up working perfectly!

It also meant that we – the other day when I did a Rebus code camp with the local .NET user group, and some people had trouble making the MSMQ of their ridiculously locked-down work laptops talk to my MSMQ – could switch from MSMQ to Rabbit by going from

to

and everything would work, even for those who were struggling with their frigid laptops… pretty cool, huh?

There are still differences, however – in the next post I’ll go through some of the reasons why you might choose one over the other…

Can your system withstand the process chaos monkey?

As part of my efforts to constantly improve and harden Rebus, I have recently put it through some testing that should reveal any weaknesses around how it handles transactions in the face of failure.

My most recent test involved a client, that would send a number of sequentially numbered messages to a FragileProcessor, that would process these messages and save the contained number in an SQL Server – and in doing this, FragileProcessor would constantly crash and burn!

To help me do this, I devised a simple application called ProcessChaosMonkey, containing this evil beast (whose name is inspired – a.k.a. shamelessly ripped off – of Netflix’s chaos monkey):

which I would then run like this: ProcessChaosMonkey.exe FragileProcessor, causing the FragileProcessor process to be constantly shot down at random times.

As FragileProcessor is a Windows Service, I configured it to recover immediately upon crashing. This way, I could leave “my system” running, processing thousands and thousands of messages, and then I could come back and see if messages had been dropped. This way of planning for failure is in my opinion the only way to go if you want to build systems, that can’t fail. Do you think your system can withstand ProcessChaosMonkey?

Oh, and if you’re interested in how the most recent testing went, stay tuned for the next post where I’ll talk a little bit about delivery guarantees…

Unit testing sagas with Rebus

Ok, so now we have taken a look at how we could unit test a simple Rebus message handler, and that looked pretty simple. When unit testing sagas, however, there’s a bit more to it because of the correlation “magic” going on behind the scenes. Therefore, to help unit testing sagas, Rebus comes with a SagaFixture<TSagaData> which you can use to wrap your saga handler while testing.

I guess an example is in order 🙂

Consider this simple saga that is meant to be put in front of a call to an external web service where the actual web service call is being made somewhere else in a service that allows for using simple request/reply with any correlation ID:

Let’s say I want to punish this saga in a unit test. A way to do this is to attack the problem like when unit testing an ordinary message handler – but then a good portion of the logic under test is not tested, namely all of the correlation logic going on behind the scenes. Which is why Rebus has the SagaFixture<TSagaData> thing… check this out:

As you can see, the saga fixture can be used to “host” your saga handler during testing, and it will take care of properly dispatching messages to it, respecting the saga’s correlation setup. Therefore, after handing over the saga to the fixture, you should not touch the saga handler again in that test.

If you want to start out in your test with some premade home-baked saga data, you can pass it to the saga fixture as a second constructor argument like so:

and as you can see, you can access all “live” saga data via fixture.AvailableSagaData, and you can access “dead” saga data as well via fixture.DeletedSagaData.

If you’re interested in logging stuff, failing in case of uncorrelated messages, etc., there’s a couple of events on the saga fixture that you can use: CorrelatedWithExistingSagaData, CreatedNewSagaData, CouldNotCorrelate – so in order to fail in all cases where a message would have been ignored because it could not be correlated, and the message is not allowed to initiate a new saga data instance, you could do something like this:

Nifty, huh?

Unit testing handlers with Rebus

…is pretty easy, because message handlers are ordinary classes free of dependencies – i.e. you implement the appropriate Handle methods, but you don’t have to derive your class off of some base class, and you can have your own dependencies injected, so you’re free to mock everything if you feel like it.

There’s one thing, though: The IBus interface, which you’ll most likely need some time, is kind of clunky to mock with mocking libraries like Moq, Rhino Mocks, NSubstitute, FakeItEasy [insert today’s hip mocking library here] – especially if you’re testing the AAA way of writing your unit tests.

Let’s take a look at a simple handler, whose responsibility is to subscribe to the PurchaseRecorded event and, for each debtor involved in the purchased mortgage deed, ensure that a process is kicked off that subscribes that debtor to an SSN-based address update service provided by the Danish Central Person Registry:

Right, so now I want my test to capture the fact that the incoming event gives rise to multiple messages that the service sends to itself, one for each debtor. My test might look somewhat like this (shown with the classic Rhino Mocks syntax):

It doesn’t require that much imagination to see how the asserts can become completely unreadable if the sent messages contain more than a few fields… which is why Rebus has a FakeBus in the Rebus.Testing namespace!

Check this out:

The example above is quite simple, so it might not be that apparent – but the real force of FakeBus is that just stores all messages that are sent, sent to self, published, replied, etc., also it also stores any headers that may have been added. This way, during testing, you can easily get access to the actually sent messages and inspect whether their information is as expected.

Slides from my Rebus talks

I have now established firmly, by giving 3 user group talks about Rebus within a relatively short period, that my throat and voice are made to STFU and code. So, I will do just that, and also make available the slides from the talks.

Actually, the slides will be the most recent deck because it’s the best and most up to date. And to the ONUG and CNUG folks, I should say this: “If you don’t recognize the third code example, it’s not (necessarily) because you were sleeping towards the end of my talk – it’s because I decided to change the third sample into something that closes the loop, and brings us back to the original problem that was the outset of the entire talk.”

The code for the (new) third sample is available on GitHub.

You can browse the slides here if you want, or you can download the PDF.

Upcoming user group talks about Rebus

Now that Rebus is gaining traction in a few places, and I have finally actually started using it myself, a little bit of active proselytization is in order – therefore, I am lucky that the Danish .NET user groups have accepted to let me come and talk about it, which is a great honor.

First, on August the 8th, I’ll visit Odense .NET User Group. Information about the Rebus event in Odense and registration is on Conferize.

Then on August 23rd, I’ll go visit Copenhagen .NET User Group – information about the Rebus event in Copenhagen and registration is on Eventbrite.

And finally, on August 29th, I’ll go visit the local .NET user group, Aarhus .NET User Group – information about the Rebus event in Aarhus and registration is on Facebook.

Please note that all of the presentations will be done in Danish and C#.

Cool way to set message expiration with Rebus

If you want to be more declarative about setting the expiration timeout on your messages, you can define your own [TimeToBeReceived(...)] attribute, and then take advantage of Rebus’ hooks to detect the attribute on sent messages and set the appropriate header.

E.g. define the attribute in your message assembly like so:

You can think of this attribute as a vital piece of business knowledge: “The information contained within this message is irrelevant after the specified time has elapsed.” Therefore, it definitely belongs somewhere within your code. You can of course argue whether it belongs within the message assembly, but there’s nothing in the snippet above that prevents you from passing MyBusinessConstants.SomeSpecificKindOfInformationMaxValidFor as the value, and then let the MyBusinessConstants be defined in another assembly. And then you could argue some more, but the point is that you get to avoid Rebus references in your message assemblies. Period.

And then you can detect it whenever Rebus sends a message by setting up a listener on the MessageSent event of the advanced bus’ Events.MessageSent event by using the event hooks configuration API, like so:

which will then allow you to apply message expiration timeouts like this:

while keeping your messages assembly clean from Rebus stuff.

Please note, however, that the APIs shown here are only available in Rebus 0.13.0-alpha and forward, as this version contains the new stuff with the ability to set up hooks with the configuration API.