When you’re testing your code, you’re most likely doing different types of tests – e.g. you might be doing real unit testing, integration testing, etc. In Rebus, we had the need to do contract tests[1. I’m not aware if there’s a more established word for this kind of black box test.], which is a test that ideally runs on multiple implementations of some interface, verifying that each implementation – given various preconditions in the form of differing setups etc – is cabable of satisfying a contract in the form of an interface.
In other words, for each implementation of a given interface, answer this question: Can the implementation behave in the same way in regards to the behavior we care about.
I’m aware of Grensesnitt, which Greg Young authored some time ago – and that is exactly what I’m after, only without the depending on an additional library, without the automatic stuff, etc. I just wanted to do it manually.
So, at first I wrote a lot of tedious code that would execute multiple tests for each test case, which was really clunky and tedious and would yield bad error messages when tests would fail, etc. Luckily, Asger Hallas brought my attention to the fact that a much more appropriate mechanism was already present in NUnit that would enable the thing that I wanted.
It turned out that NUnit’s [TestFixture] attribute accepts a Type as an argument, i.e. you can do this: [TestFixture(typeof(SomeType))], which will cause the test fixture type, which should be generic, to be closed with the given type. E.g. like so:
|
1 2 3 4 5 6 7 8 9 10 |
[TestFixture(typeof(SomeClassSomewhere))] public class MyTestFixture<T> { [Test] public void CheckType() { // so, what to do with T in here? Assert.That(typeof(T), Is.EqualTo(typeof(SomeClassSomewhere))); } } |
Now, if we add a generic constraint to the fixture’s type parameter, we can require that the type is a factory and can be newed up:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
[TestFixture(typeof(SomeClassSomewhere))] public class MyTestFixture<T> where T : ISomeParticularInterface, new() { [Test] public void CheckType() { // oh wow, now we can do stuff with T ISomeParticularInterface t = new T(); t.CallSomethingFromTheInterface(); } } |
See where this is going? Now one single test fixture can be closed with multiple factory types, where each factory type is capable of creating an instance of one particular implementation of something, that should be tested. E.g. in Rebus, we can test that multiple implementations of the ISendMessages and IReceiveMessages work as expected by creating a factory interface like this:
|
1 2 3 4 5 |
public interface ITransportFactory { Tuple<ISendMessages, IReceiveMessages> Create(); void CleanUp(); } |
where an implementation might look like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
public class RabbitMqTransportFactory : ITransportFactory { RabbitMqMessageQueue sender; RabbitMqMessageQueue receiver; public Tuple<ISendMessages, IReceiveMessages> Create() { sender = new RabbitMqMessageQueue(RabbitMqFixtureBase.ConnectionString, "tests.contracts.sender", "tests.contracts.sender.error"); receiver = new RabbitMqMessageQueue(RabbitMqFixtureBase.ConnectionString, "tests.contracts.receiver", "tests.contracts.receiver.error"); return new Tuple<ISendMessages, IReceiveMessages>(sender, receiver); } public void CleanUp() { sender.Dispose(); receiver.Dispose(); } } |
and one particular test fixture might look like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
[TestFixture(typeof(MsmqTransportFactory))] [TestFixture(typeof(RabbitMqTransportFactory)), Category(TestCategories.Rabbit)] public class TestSendAndReceive<TFactory> : FixtureBase where TFactory : ITransportFactory, new() { static readonly TimeSpan MaximumExpectedQueueLatency = TimeSpan.FromMilliseconds(300); TFactory factory; ISendMessages sender; IReceiveMessages receiver; protected override void DoSetUp() { factory = new TFactory(); var transports = factory.Create(); sender = transports.Item1; receiver = transports.Item2; } protected override void DoTearDown() { factory.CleanUp(); } /// ....and then there's test cases down here working on sender and receiver |
The advantage is that it’s much easier to see if new implementations of Rebus’ abstractions can satisfy the required contracts – and with the way it’s implemented now, various implementations can have different setup and teardown code, and their fixtures can belong to a suitable NUnit category, allowing Rebus’ build servers to run only tests where the required infrastructure is in place.
The problem with this (and what gs originally solved is) I have 300 fixtures … I add a new implementor and now need to go add attributes all over the place. What if I forget to add one?
Yeah, I get that there are scenarios where automating that part is beneficial. In Rebus, however, I want to – in a very manual process – consider each contract (or an apect thereof) individually, and then apply the [TestFixture(typeof(SomeSpecificImplementationFactory))] if the contract or aspect should be adhered to by that specific implementation.
E.g. saga persisters – they basically come in two kinds, 1) those who can correlate a message with multiple saga instances and commit the bunch atomically, and 2) those who can only commit one instance atomically and therefore may not correlate incoming messages with multiple saga instances, which in turn can only be implemented if the underlying data store can sustain a unique constraint on the correlation ID in question.
Both are saga persisters, and they almost satisfy the same invariants, except e.g. the one mentioned here.
Should they be the same interface then?
I can see how I am contradicting myself here 🙂 I say “it’s the same contract, but they’re different”, but the fact that they differ implies that they’re not the same contract. I’d be thrilled if you could help me get a better take on this!
I have a bus, it’s called Rebus, it’s very similar to NServiceBus, MassTransit, and Rhino ESB. It needs to store sagas somewhere, and the bus gets that ability provided via the IStoreSagaData abstraction, which is currently implemented in a MSSQL, RavenDB, and a MongoDB flavor.
Now, due to the way the different databases have different properties, it’s impossible to achieve the exact same behavior in all three databases. This becomes apparent when an incoming message gets processed, and the bus needs to correlate a field of the incoming message with a field in an existing saga.
With MongoDB, the bus can only load one saga instance, because the DB can only perform one update per transaction – because if I were to try performing multiple updates and an optimistic locking exception occurred on the second update, then there’d be no way to roll back the first update.
MongoDB can help me enforce this constraint though, because it allows me to add a unique constraint on a property, thus guaranteeing that at most one document can be correlated with each incoming message.
RavenDB has no way of enforcing uniqueness on arbitrary properties in documents, it does however provide the ability to modify multiple documents in a transaction with optimistic concurrency and all, which MongoDB didn’t.
To sum it up: Implementors of IStoreSagaData must either a) throw immediately if I try to save a saga with a correlation property value that already exists, or b) be able to modify several sagas in a single transaction.
So: Implementors share the same purpose, and they achieve it with slightly differing properties where each implementation should just guarantee that it compensates for its own shortcomings.
Well, actually I might have talked myself into something here – maybe I’m saying that my tests should be written on a form that verifies exactly what I wrote in the previous paragraph – that implementations should guarantee that they compensate for their own shortcomings.
Wow, that was a long reply 🙂 thanks for your comments!