Using Windsor as an auto-mocking container

2011-10-05 by mookid8000 Leave a comment

One of my former colleagues blogged about using AutoFixture as an auto-mocking container the other day, which got me thinking about auto-mocking with Windsor.

Although I’ve never used auto-mocking myself, a few months ago, I answered a question on StackOverflow, hinting at how auto-mocking could be accomplished with Castle Windsor. I didn’t provide any example code though, so this post will show a more complete solution on how auto-mocking can be implemented with Windsor and Rhino Mocks[1. I realize that all the cool kids are using other mocking libraries these days. I’m still using Rhino though, but it should be a fairly trivial task to “port” this solution to another mocking library.].

A lazy component loader to generate the mocks

First, I create a simple ILazyComponentLoader that lazily registers a Rhino Mocks instance when a particular service is requested – like so:

public class AutoMockingLazyComponentLoader : ILazyComponentLoader
{
    public IRegistration Load(string key, Type service, IDictionary arguments)
    {
        return Component.For(service).Instance(MockRepository.GenerateMock(service, new Type[0]));
    }
}

public class AutoMockingLazyComponentLoader : ILazyComponentLoader

{

public IRegistration Load(string key, Type service, IDictionary arguments)

{

return Component.For(service).Instance(MockRepository.GenerateMock(service, new Type[0]));

}

– real simple. Windsor’s default lifestyle is singleton, which ensures that subsequent calls for the same service will give me the same mock instance.

Test fixture base class

Then I create a test fixture base class, which is supposed to act as exactly that: a fixture for the SUT:

public abstract class AutoMockingFixtureFor<TAppService>
{
    IWindsorContainer container;

    protected TAppService service;

    [SetUp]
    public void SetUp()
    {
        container = new WindsorContainer()
            .Register(Component.For<ILazyComponentLoader>().ImplementedBy<AutoMockingLazyComponentLoader>(),
                      Component.For<TAppService>());

        // build sut and inject mocks for all dependencies
        service = container.Resolve<TAppService>();

        DoSetUp();
    }

    protected virtual void DoSetUp() { }

    [TearDown]
    public void TearDown()
    {
        DoTearDown();

        container.Dispose()
    }

    protected virtual void DoTearDown() { }

    protected TDependency Dep<TDependency>()
    {
        return container.Resolve<TDependency>();
    }

    protected TMock Mock<TMock>() where TMock : class
    {
        return MockRepository.GenerateMock<TMock>();
    }
}

public abstract class AutoMockingFixtureFor<TAppService>

{

IWindsorContainer container;

protected TAppService service;

[SetUp]

public void SetUp()

{

container = new WindsorContainer()

.Register(Component.For<ILazyComponentLoader>().ImplementedBy<AutoMockingLazyComponentLoader>(),

Component.For<TAppService>());

// build sut and inject mocks for all dependencies

service = container.Resolve<TAppService>();

DoSetUp();

}

protected virtual void DoSetUp() { }

[TearDown]

public void TearDown()

{

DoTearDown();

container.Dispose()

}

protected virtual void DoTearDown() { }

protected TDependency Dep<TDependency>()

{

return container.Resolve<TDependency>();

}

protected TMock Mock<TMock>() where TMock : class

{

return MockRepository.GenerateMock<TMock>();

}

As you can see, my SetUp method creates a new WindsorContainer, registering nothing but my AutoMockingLazyComponentLoader and TAppService – the SUT type. It then uses the container to instantiate the SUT, storing it away in a protected instance variable for test cases to work on.

I included DoSetUp and DoTearDown methods for my test fixtures to override in case they need to – but in most cases, they should not be used because of the coupling they introduce between test cases.

Lastly, I have two methods: Dep, which allows me to access an injected service type (short for “dependency”), and Mock which allows me to generate a new mock object in case I need to mock something other than my SUT’s dependencies.

How to write a new test fixture

As a result of this, a test fixture for something called HandleUpdateDisturbanceForecast is reduced to something like this:

[TestFixture]
public class TestOfHandleUpdateDisturbanceForecast : AutoMockingFixtureFor<HandleUpdateDisturbanceForecast>
{
    [Test]
    public void UpdatesDriverWithReceivedObservableForecast()
    {
        // Arrange
        var now = 19.March(1979).At(12.Hours());
        var localUnit = new GenericDriver();
        Dep<IProvidesSiteToStartUp>().Stub(p => p.GetLocalUnitByAlias("ali")).Return(localUnit);

        // Act
        service.Handle(new UpdateDisturbanceForecast
                            {
                                LocalUnitAlias = "ali",
                                DisturbanceForecast = new List<DisturbancePlanEntry>
                                                            {
                                                                new DisturbancePlanEntry {Time = now, Disturbance = 2},
                                                            }
                            });

        // Assert
        var forecast = localUnit.DisturbanceForecast;
        Assert.AreEqual(1, forecast.Count);
        Assert.AreEqual(now, forecast[0].Time);
        Assert.AreEqual(2, forecast[0].Value);
    }
}

[TestFixture]

public class TestOfHandleUpdateDisturbanceForecast : AutoMockingFixtureFor<HandleUpdateDisturbanceForecast>

{

[Test]

public void UpdatesDriverWithReceivedObservableForecast()

{

// Arrange

var now = 19.March(1979).At(12.Hours());

var localUnit = new GenericDriver();

Dep<IProvidesSiteToStartUp>().Stub(p => p.GetLocalUnitByAlias("ali")).Return(localUnit);

// Act

service.Handle(new UpdateDisturbanceForecast

{

LocalUnitAlias = "ali",

DisturbanceForecast = new List<DisturbancePlanEntry>

{

new DisturbancePlanEntry {Time = now, Disturbance = 2},

}

});

// Assert

var forecast = localUnit.DisturbanceForecast;

Assert.AreEqual(1, forecast.Count);

Assert.AreEqual(now, forecast[0].Time);

Assert.AreEqual(2, forecast[0].Value);

}

– which I think looks pretty slick. And yes, this is an actual test case from something we’re building – doesn’t matter what it’s doing, just wanted to show an actual example.

It’s not like I’m saving a huge amount of coding here – I usually only instantiate my SUT in the SetUp method of my test fixtures, but I like how the “fixed style” of AutoMockingFixtureFor encourages application services to follow a pattern that makes for easy IoC and testing. And the fixture is relieved of almost all clutter that does not directly relate to the thing being tested.

Castle Windsor resolution logic: Cool handler filter that orders components

2011-08-15 by mookid8000 Leave a comment

As you can see from my previous post on handler filters, an IHandlersFilter will have a chance to do stuff to the array of handlers that would otherwise be used untouched during a call to ResolveAll ( ResolveAll is also what happens behind the scenes when a component relies on CollectionResolver to supply a collection of components for it to use).

Now, what is left is to implement some kind of logic that does something to the handlers array. First thing that comes to my mind is ordering the handlers!

Ordering the handlers is very relevant e.g. in scenarios where tasks are modeled as a series of steps that should be executed in a consistent fashion. An example could be a component ( TaskProcessor) that gets a collection of implementations of some task-like interface injected ( IEnumerable<ITask>).

Now, to ensure that this list of tasks is properly ordered, I’d like to specify a few hints on how they should be ordered. Not too little, because otherwise stuff would not work, but I also don’t want to be overly explicit on how the tasks should be ordered.

So I came up with a solution based on two attributes: ExecutesBeforeAttribute and ExecutesAfterAttribute, allowing types to position themselves in relation to other types they know about, and all other types to stay oblivious of what is going on.

One possible usage scenario could look like this, assuming we have modeled some kind of money transfer as a series of steps, each implementing ITask:

public interface ITask
{
    void Execute();
}

public class ExecutePayment : ITask { ... }

public class ValidateCreditCards : ITask { ... }

public class ValidateDebitAccountBalance : ITask { ... }

public class ReportWarnings : ITask { ... }

public class GenerateReceipt : ITask { ... }

public interface ITask

{

void Execute();

}

public class ExecutePayment : ITask { ... }

public class ValidateCreditCards : ITask { ... }

public class ValidateDebitAccountBalance : ITask { ... }

public class ReportWarnings : ITask { ... }

public class GenerateReceipt : ITask { ... }

and then a simplistic executor:

public class TaskExecutor
{
    readonly IEnumerable<ITask> tasks;

    public TaskExecutor(IEnumerable<ITask> tasks)
    {
        this.tasks = tasks;
    }

    public void ExecuteTasks()
    {
        foreach(var task in tasks)
        {
            task.Execute();
        }
    }
}

public class TaskExecutor

{

readonly IEnumerable<ITask> tasks;

public TaskExecutor(IEnumerable<ITask> tasks)

{

this.tasks = tasks;

}

public void ExecuteTasks()

{

foreach(var task in tasks)

{

task.Execute();

}

Now, carrying out a money transfer would consist of having the executor run through this list of discrete tasks, but it’s probably important that the validation occurs before the actual payment, and it probably doesn’t make sense to generate reports before the payment. Therefore, let’s decorate some of the steps:

public class ExecutePayment : ITask { ... }

[ExecutesBefore(typeof(ExecutePayment))]
public class ValidateCreditCards : ITask { ... }

[ExecutesBefore(typeof(ExecutePayment))]
public class ValidateDebitAccountBalance : ITask { ... }

[ExecutesAfter(typeof(ExecutePayment))]
public class ReportWarnings : ITask { ... }

[ExecutesAfter(typeof(ExecutePayment))]
public class GenerateReceipt : ITask { ... }

public class ExecutePayment : ITask { ... }

[ExecutesBefore(typeof(ExecutePayment))]

public class ValidateCreditCards : ITask { ... }

[ExecutesBefore(typeof(ExecutePayment))]

public class ValidateDebitAccountBalance : ITask { ... }

[ExecutesAfter(typeof(ExecutePayment))]

public class ReportWarnings : ITask { ... }

[ExecutesAfter(typeof(ExecutePayment))]

public class GenerateReceipt : ITask { ... }

and then make sure we respect them:

var container = new WindsorContainer();

// allow collections to be resolved
container.Kernel.Resolver.AddSubResolver(new CollectionResolver(container.Kernel));

// this is the new thing!
container.Kernel.AddHandlersFilter(new RespectOrderDirectivesHandlersFilter(typeof(ITask)));

container.Register(AllTypes.FromThisAssembly().BasedOn<ITask>().WithService.Base(),
                    Component.For<TaskExecutor>());

var container = new WindsorContainer();

// allow collections to be resolved

container.Kernel.Resolver.AddSubResolver(new CollectionResolver(container.Kernel));

// this is the new thing!

container.Kernel.AddHandlersFilter(new RespectOrderDirectivesHandlersFilter(typeof(ITask)));

container.Register(AllTypes.FromThisAssembly().BasedOn<ITask>().WithService.Base(),

Component.For<TaskExecutor>());

Now, when the container builds a TaskExecutor, it will inject the ITask implementations in the order specified by the attributes. Inside RespectOrderDirectivesHandlersFilter, there’s a HandlerSorter, which is the implementation of the actual sorting algorithm. It builds a directed graph out of components and their dependents, and then it orders the components so that they will be traversed in a way that respects the order specified by the attributes. I’m not an expert on graph algorithms, so I don’t know if my implementation is really really stupid and slow – but I know that a) it works, and b) the ordering is cached, so the algorithm will only run once per set of handlers.

If you want to know more, or perhaps put RespectOrderDirectivesHandlersFilter to use, you can visit the RespectOrderDirectivesHandlersFilter page on GitHub.

Castle Windsor resolution logic: Handler filters

2011-08-10 by mookid8000 Leave a comment

One of the features I’m looking forward to in the upcoming Castle Windsor 3.0 (“Wawel”) release is handler filters! I’m especially looking forward to this feature because it solves a problem I have had quite a few times now, and also because it’s a feature that I’ve contributed to the Windsor project.

Please note that handler filter are NOT available before Windsor 3.0, which will probably be out sometime around middle August 2011.

As you can see, handler selectors is a hook in Resolve that sorts out the situation by choosing one handler among all the available handlers. Pretty much related to this, are handler filters, which is a hook in ResolveAll, that sorts out the situation by choosing and ordering several handlers from all the available handlers. Let’s take a look at a tiny example….

Example: Task processing pipeline

A golden use case for this feature is when you rely on CollectionResolver to inject a collection for you, e.g. in chain of responsibility-like scenarios like so:

public class TaskProcessor
{
    readonly IEnumerable<ITask> tasks;
    readonly IErrorReporter errorReporter;

    public TaskProcessor(IEnumerable<ITask> tasks, IErrorReporter errorReporter)
    {
        this.tasks = tasks;
        this.errorReporter = errorReporter;
    }

    public void ProcessTasks()
    {
        foreach(var task in tasks)
        {
            try
            {
                task.DoIt();
            }
            catch(SomeDomainException ex)
            {
                errorReporter.ReportErrorProcessingTask(task, ex);
                break;
            }
        }
    }
}

public class TaskProcessor

{

readonly IEnumerable<ITask> tasks;

readonly IErrorReporter errorReporter;

public TaskProcessor(IEnumerable<ITask> tasks, IErrorReporter errorReporter)

{

this.tasks = tasks;

this.errorReporter = errorReporter;

}

public void ProcessTasks()

{

foreach(var task in tasks)

{

try

{

task.DoIt();

}

catch(SomeDomainException ex)

{

errorReporter.ReportErrorProcessingTask(task, ex);

break;

}

Usually, when doing this kind of task processing, you care about the order in which the tasks are processed. Let’s pretend we have some tasks:

public class FinishTheJob : ITask
{
    public void DoIt()
    {
        Console.WriteLine("Finishing stuff");
    }
}

public class PrepareSomething : ITask
{
    public void DoIt()
    {
        Console.WriteLine("Preparing stuff");
    }
}

public class CarryItOut : ITask
{
    public void DoIt()
    {
        Console.WriteLine("Carrying out some important logic");
    }
}

public class FinishTheJob : ITask

{

public void DoIt()

{

Console.WriteLine("Finishing stuff");

}

public class PrepareSomething : ITask

{

public void DoIt()

{

Console.WriteLine("Preparing stuff");

}

public class CarryItOut : ITask

{

public void DoIt()

{

Console.WriteLine("Carrying out some important logic");

}

and the tasks are registered “dynamically” (allowing us to add new tasks just by dropping new implementations of ITask into the project), like so:

container.Register(AllTypes.FromThisAssembly().BasedOn<ITask>().WithService.Base());

1	container.Register(AllTypes.FromThisAssembly().BasedOn<ITask>().WithService.Base());

Now, how do we make sure that the PrepareSomething and FinishTheJob tasks are run at the right time, i.e. before and after CarryItOut? Handler filters to the rescue!

Let’s register a handler filter like so:

container.Kernel.AddHandlersFilter(new TaskHandlersFilter());

1	container.Kernel.AddHandlersFilter(new TaskHandlersFilter());

and the filter is implemented like this:

class TaskHandlersFilter : IHandlersFilter
{
    readonly Dictionary<Type, int> sortOrder = new Dictionary<Type, int>
                                                   {
                                                       {typeof (PrepareSomething), 1},
                                                       {typeof (CarryItOut), 2},
                                                       {typeof (FinishTheJob), 3},
                                                   };

    public bool HasOpinionAbout(Type service)
    {
        return service == typeof(ITask);
    }

    public IHandler[] SelectHandlers(Type service, IHandler[] handlers)
    {
        // come up with some way of ordering implementations here
        // (cool solution coming up in the next post... ;))
        return handlers
            .OrderBy(h => sortOrder[h.ComponentModel.Implementation])
            .ToArray();
    }
}

class TaskHandlersFilter : IHandlersFilter

{

readonly Dictionary<Type, int> sortOrder = new Dictionary<Type, int>

{

{typeof (PrepareSomething), 1},

{typeof (CarryItOut), 2},

{typeof (FinishTheJob), 3},

};

public bool HasOpinionAbout(Type service)

{

return service == typeof(ITask);

}

public IHandler[] SelectHandlers(Type service, IHandler[] handlers)

{

// come up with some way of ordering implementations here

// (cool solution coming up in the next post... ;))

return handlers

.OrderBy(h => sortOrder[h.ComponentModel.Implementation])

.ToArray();

}

Now, when my program does this:

foreach(var task in container.ResolveAll<ITask>())
{
    task.DoIt();
}

foreach(var task in container.ResolveAll<ITask>())

{

task.DoIt();

}

I get this output:

Preparing stuff
Carrying out some important logic
Finishing stuff

Preparing stuff

Carrying out some important logic

Finishing stuff

just as expected, which in turn means that CollectionResolver’s call to ResolveAll will also yield an ordered list of tasks.

One could also imagine that only certain handlers were returned, thus allowing tenants in multi-tenant scenarios to have different task processing pipelines.

Nifty, huh? In the next post, I will show a simple yet sophisticated way of ordering the handlers in the SelectHandlers method of a handler filter.

Castle Windsor resolution logic: Handler selectors

2011-08-08 by mookid8000 Leave a comment

Do you know what handler selectors are? It’s a way for you to register some logic in Windsor that will choose among all the available handlers when some particular component is resolved.

E.g. if you have a multi-tenant application with multiple implementations of IUserNotifier, let’s call them NotifyUserBySms and NotifyUserByEmail, and you need to make sure that one particular tenant whose employees are equipped with smartphones get notified by email, you can let a handler selector make this selection transparent to the rest of your application by adding a handler selector, like so:

class UserNotificationHandlerSelector : IHandlerSelector
{
    public bool HasOpinionAbout(string key, Type service)
    {
        return service == typeof (IUserNotifier);
    }

    public IHandler SelectHandler(string key, Type service, IHandler[] handlers)
    {
        // somehow determine whether employees of current tenant can receive emails
        Console.WriteLine("Does the current tenant' s employees have smartphones? (y/n)");

        switch (char.ToLower(Console.ReadKey().KeyChar))
        {
            case 'y':
                return handlers.Single(h => h.ComponentModel.Implementation == typeof (NotifyUserByEmail));
            default:
                return handlers.First(h => h.ComponentModel.Implementation != typeof (NotifyUserByEmail));
        }
    }
}

class UserNotificationHandlerSelector : IHandlerSelector

{

public bool HasOpinionAbout(string key, Type service)

{

return service == typeof (IUserNotifier);

}

public IHandler SelectHandler(string key, Type service, IHandler[] handlers)

{

// somehow determine whether employees of current tenant can receive emails

Console.WriteLine("Does the current tenant' s employees have smartphones? (y/n)");

switch (char.ToLower(Console.ReadKey().KeyChar))

{

case 'y':

return handlers.Single(h => h.ComponentModel.Implementation == typeof (NotifyUserByEmail));

default:

return handlers.First(h => h.ComponentModel.Implementation != typeof (NotifyUserByEmail));

}

thus making it possible for all other services to depend on IUserNotifier, not caring under which tenant’s context they’re running.

As you can see, this handler selector only has an opinion when an IUserNotifier is being resolved. And then, when its SelectHandler method is invoked, it gets to choose among all assignable handlers for that particular service type, effectively providing a way to make sure that some particular implementation is used, given some condition (here simulated by some Console.ReadKey() action).

Nifty huh? Only sad thing though, is that handler selectors are not pulled from the container – you have to supply an instance to the kernel, so if you need some services in the selector you need to pull them manually from the container. This can of course be overcome by supplying the handler selector with a reference to the container, allowing it to Resolve stuff at will – just remember – as always – to Release what you Resolve.

This was a short introduction to handler selectors – next post will be about something new, namely handler filters which is a new feature in Windsor 3.0.

Frictionless Persistence in .NET with MongoDB is available on Channel 9

2011-06-24 by mookid8000 1 Comment

Microsoft were cool enough to record my talk at this year’s Goto Cph, even though the talk was on the Cloud and NoSQL track.

You can watch it here.

Afterwards, go check out all the other great presentations – it’s so cool they’ re made available!

On the bus with MassTransit #1

2011-06-03 by mookid8000 4 Comments

Update: If you want to read about Topshelf, please go to the updated guide to Topshelf, which covers how to get started with Topshelf 3

This is the first post in a series of posts on the open source service bus implementation, MassTransit. In order to push myself through learning it, I will do these posts as I go and document stuff I learn on the way.

This first post will be about one thing, that you’ll probably end up doing quite a few times while building your awesome messaging-based distributed system: Creating a service.

Check out TopShelf and build it

I’ve described how to use TopShelf before, but a lot has changed since then. TopShelf, at the time of writing, is now in version 2.2.1.0, and since 2.0 it has a feature called “shelving”, which is the ability to host any number of services beneath one Windows Service – just like IIS hosts web apps. This is pretty awesome, but today I’ll just show “the old API”, which IMO is still pretty slick!

First, make sure you have a working Ruby installation with Albacore ( gem install albacore). You’ll need these to build Topshelf.

Next, git clone git://github.com/phatboyg/Topshelf.git and build.bat. That should leave a bunch of binaries in the NET35 and NET40 folders beneath build_output. This is open source, baby!

Put your app on the TopShelf

Create a new “Console Application” and make sure to target the full .NET 4 framework (and not that bastard .NET 4 Client Profile thingie, which is default…)

Now add a reference to Topshelf.dll and log4net.dll. Punch in something like this:

class Program
{
    static void Main()
    {
        HostFactory.Run(x =>
        {
            x.SetDescription("MyBackend-description");
            x.SetDisplayName("MyBackend-display-name");
            x.SetServiceName("MyBackend-service-name");
        });
    }
}

class Program

{

static void Main()

{

HostFactory.Run(x =>

{

x.SetDescription("MyBackend-description");

x.SetDisplayName("MyBackend-display-name");

x.SetServiceName("MyBackend-service-name");

});

}

This is all the necessary stuff to create a fully functional program, that can be run and debugged by running the resulting .exe – either from the command line, or with F5 inside Visual Studio.

Moreover, it can be installed as a Windows Service by going to the command line (as an administrator!) and executing NameOfYourApp.exe install. On my machine, I got this output:

C:\temp\TransitDemo\Backend\bin\Debug>Backend.exe  install

Running a transacted installation.

Beginning the Install phase of the installation.
Installing service MyBackend-service-name..
Service MyBackend-service-name has been successfully installed.
Creating EventLog source MyBackend-service-name in log Application...

The Install phase  completed successfully, and the Commit phase is beginning.

The Commit phase completed successfully.

The transacted install has completed.

C:\temp\TransitDemo\Backend\bin\Debug>Backend.exe install

Running a transacted installation.

Beginning the Install phase of the installation.

Installing service MyBackend-service-name..

Service MyBackend-service-name has been successfully installed.

Creating EventLog source MyBackend-service-name in log Application...

The Install phase completed successfully, and the Commit phase is beginning.

The Commit phase completed successfully.

The transacted install has completed.

Now I can net start MyBackend-service-name and net stop MyBackend-service-name on the command line, or I can go to Windows’ service control snap-in and pull the usual levers to control it. Nifty!

Now I have a perfectly functioning Windows Service that does nothing at all! Let’s output some stuff to see where we’ll put our code in a short while. Try extending the x => { ... } lambda to make the program look like this:

class Program
{
    static void Main()
    {
        HostFactory.Run(x =>
        {
            x.Service<MyBackend>(c =>
            {
                c.SetServiceName("MyBackend");
                c.ConstructUsing(() => new MyBackend());
                c.WhenStarted(d => d.StartMyBackend());
                c.WhenStopped(d => d.StopMyBackend());
            });

            x.SetDescription("MyBackend-description");
            x.SetDisplayName("MyBackend-display-name");
            x.SetServiceName("MyBackend-service-name");
        });
    }
}

class Program

{

static void Main()

{

HostFactory.Run(x =>

{

x.Service<MyBackend>(c =>

{

c.SetServiceName("MyBackend");

c.ConstructUsing(() => new MyBackend());

c.WhenStarted(d => d.StartMyBackend());

c.WhenStopped(d => d.StopMyBackend());

});

x.SetDescription("MyBackend-description");

x.SetDisplayName("MyBackend-display-name");

x.SetServiceName("MyBackend-service-name");

});

}

and a class that looks something like this:

class MyBackend
{
    public void StartMyBackend()
    {
        Console.WriteLine("Started!");
    }

    public void StopMyBackend()
    {
        Console.WriteLine("Stopped!");
    }
}

class MyBackend

{

public void StartMyBackend()

{

Console.WriteLine("Started!");

}

public void StopMyBackend()

{

Console.WriteLine("Stopped!");

}

Now, when I press F5, wait a little, and press Ctrl + C, I get this output on the console:

Started!
Stopped!
Press any key to continue . . .

Started!

Stopped!

Press any key to continue . . .

Awesome, huh?

I really like how TopShelf now uses a Func<TService> to construct its service instance via ConstructUsing, instead of depending on the common service locator as the old TopShelf did. This way, I can pull the instance from an IoC container if I want.

In the next post I will take a look at how to send a message from a web application when someone tries to register as a user.

Free geek night about MongoDB

2011-05-30 by mookid8000 Leave a comment

At Goto Copenhagen this May, I gave a talk about MongoDB, which is a nifty document-oriented database that I find pretty interesting.

So, because I like to talk about MongoDB so much, I’ll give my talk again as a free Trifork geek night on Tuesday the 21th of June at the Trifork HQ in Aarhus (this time in Danish though).

If you’re a .NET person, possibly developing big enterprisey stuff and/or you’re interested in MongoDB or NoSQL in general, you should come to this one.

Tinkering with MongoDB and sharding

2011-05-12 by mookid8000 5 Comments

After my MongoDB presentation today, I was asked a few questions about MongoDB’s sharding capabilities. Like my interest so far, my talk was completely focused on the frictionless aspects of using MongoDB, so I have never tried any of the sharding and replica set configurations that MongoDB can run in.

That has got to end!

So, let’s try spinning up the simplest possible sharding scenario that we can think of: Two durable MongoDB instances on port 27017 and 27018 with one collection sharded across them:

mkdir c:\data\db1
mongod --shardsvr --port 27017 --journal --dbpath c:\data\db1

1 2	mkdir c:\data\db1 mongod --shardsvr --port 27017 --journal --dbpath c:\data\db1

mkdir c:\data\db2
mongod --shardsvr --port 27018 --journal --dbpath c:\data\db2

1 2	mkdir c:\data\db2 mongod --shardsvr --port 27018 --journal --dbpath c:\data\db2

MongoDB sharding requires that you spin up a special configuration server that stores the configuration of which shards are available – let’s spin this up on port 27019:

mkdir c:\data\configdb
mongod --configsvr --port 27019 --dbpath c:\data\configdb

1 2	mkdir c:\data\configdb mongod --configsvr --port 27019 --dbpath c:\data\configdb

– and finally, we spin up one instance of mongos on port 27020, pointing it towards the configuration server:

mongos --configdb localhost:27019 --port 27020

1	mongos --configdb localhost:27019 --port 27020

To finalize the setup, we let the configuration database know of the two shards we have started by connecting to the admin database using the Mongo shell:

C:\mongodb\bin>mongo localhost:27020
MongoDB shell version: 1.8.1
connecting to: localhost:27020/test
> use admin
switched to db admin
> db.runCommand({addShard:"localhost:27017"})
> db.runCommand({addShard:"localhost:27018"})

C:\mongodb\bin>mongo localhost:27020

MongoDB shell version: 1.8.1

connecting to: localhost:27020/test

> use admin

switched to db admin

> db.runCommand({addShard:"localhost:27017"})

> db.runCommand({addShard:"localhost:27018"})

Now, let’s see if it understood this:

> db.runCommand({listShards: 1})
{
        "shards" : [
                {
                        "_id" : "shard0000",
                        "host" : "localhost:27017"
                },
                {
                        "_id" : "shard0001",
                        "host" : "localhost:27018"
                }
        ],
        "ok" : 1
}

> db.runCommand({listShards: 1})

{

"shards" : [

{

"_id" : "shard0000",

"host" : "localhost:27017"

{

"_id" : "shard0001",

"host" : "localhost:27018"

}

"ok" : 1

}

As you can see, the config database correctly stored information about our two shards – that was easy!

Finally, we need to enable sharding for one particular database and make sure that our collection of unicorns is sharded by the name field:

> db.runCommand({enableSharding: "fairytale"})
{ "ok" : 1 }
> db.runCommand({shardCollection: "fairytale.unicorns", key: {name: 1}})
{ "collectionsharded" : "fairytale.unicorns", "ok" : 1 }

> db.runCommand({enableSharding: "fairytale"})

{ "ok" : 1 }

> db.runCommand({shardCollection: "fairytale.unicorns", key: {name: 1}})

{ "collectionsharded" : "fairytale.unicorns", "ok" : 1 }

At this point, even though the fairytale database and unicorns collection did not exist, they have been created for me, and the required index has been created for the shard key. I can verify this like so:

> use fairytale
> db.unicorns.getIndexes()
[
        {
                "name" : "_id_",
                "ns" : "fairytale.unicorns",
                "key" : {
                        "_id" : 1
                },
                "v" : 0
        },
        {
                "ns" : "fairytale.unicorns",
                "key" : {
                        "name" : 1
                },
                "name" : "name_1",
                "v" : 0
        }
]

> use fairytale

> db.unicorns.getIndexes()

[

{

"name" : "_id_",

"ns" : "fairytale.unicorns",

"key" : {

"_id" : 1

"v" : 0

{

"ns" : "fairytale.unicorns",

"key" : {

"name" : 1

"name" : "name_1",

"v" : 0

}

]

Now, let’s go to C# with mongo-csharp and hammer 10 million randomly named unicorns, each carrying a payload of up to 8 KB of fairy dust in there:

class Program
{
    static readonly Random Random = new Random();

    static void Main()
    {
        var server = MongoServer.Create("mongodb://localhost:27020");
        var database = server.GetDatabase("fairytale");
        var unicorns = database.GetCollection("unicorns");

        using (database.RequestStart())
        {
            var batch = Enumerable.Range(0, 10000000)
                .Select(_ =>
                        new
                            {
                                name = GenerateRandomUnicornName(),
                                horns = 1,
                                likes = new[] {"eggs", "beer"},
                                fairy_dust = GenerateRandomBytes(Random.Next(8192))
                            });
                
            unicorns.InsertBatch(batch);

            server.GetLastError();
        }
            
        server.Disconnect();
    }

    static byte[] GenerateRandomBytes(int length)
    {
        var bytes = new byte[length];
        Random.NextBytes(bytes);
        return bytes;
    }

    static string GenerateRandomUnicornName()
    {
        Func<char> charFactory = () => (char)('a' + Random.Next(25));
        var funcs = Enumerable.Repeat(charFactory, 10 + Random.Next(30));
        return new string(funcs.Select(f => f()).ToArray());
    }
}

class Program

{

static readonly Random Random = new Random();

static void Main()

{

var server = MongoServer.Create("mongodb://localhost:27020");

var database = server.GetDatabase("fairytale");

var unicorns = database.GetCollection("unicorns");

using (database.RequestStart())

{

var batch = Enumerable.Range(0, 10000000)

.Select(_ =>

new

{

name = GenerateRandomUnicornName(),

horns = 1,

likes = new[] {"eggs", "beer"},

fairy_dust = GenerateRandomBytes(Random.Next(8192))

});

unicorns.InsertBatch(batch);

server.GetLastError();

}

server.Disconnect();

}

static byte[] GenerateRandomBytes(int length)

{

var bytes = new byte[length];

Random.NextBytes(bytes);

return bytes;

}

static string GenerateRandomUnicornName()

{

Func<char> charFactory = () => (char)('a' + Random.Next(25));

var funcs = Enumerable.Repeat(charFactory, 10 + Random.Next(30));

return new string(funcs.Select(f => f()).ToArray());

}

now, let’s go to the Mongo shell and see if they’re there:

> db.unicorns.count()
10000000

1 2	> db.unicorns.count() 10000000

Great! 10 million unicorns in there. Let’s check out the disk and see if data was somehow distributed among the shards:

That seems to be pretty well balanced if you ask me. Let’s see what Mongo can tell us:

> db.printShardingStatus()
--- Sharding Status ---
  sharding version: { "_id" : 1, "version" : 3 }
  shards:
      { "_id" : "shard0000", "host" : "localhost:27017" }
      { "_id" : "shard0001", "host" : "localhost:27018" }
  databases:
        { "_id" : "admin", "partitioned" : false, "primary" : "config" }
        { "_id" : "fairytale", "partitioned" : true, "primary" : "shard0000" }
                fairytale.unicorns chunks:
                                shard0001       66
                                shard0000       65
                        too many chunksn to print, use verbose if you want to force print

> db.printShardingStatus()

--- Sharding Status ---

sharding version: { "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard0000", "host" : "localhost:27017" }

{ "_id" : "shard0001", "host" : "localhost:27018" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" }

{ "_id" : "fairytale", "partitioned" : true, "primary" : "shard0000" }

fairytale.unicorns chunks:

shard0001 66

shard0000 65

too many chunksn to print, use verbose if you want to force print

66 chunks on the 1st shard, and 65 chunks on the 2nd shard – it is in fact pretty well balanced.

Conclusion so far

It seems to be pretty easy to begin sharding data, which is perfectly in line with the usual MongoDB feeling. It’s definitely a subject I need to look more into though, so if you want to read more about it, I can really recommend Kristina Chodorow‘s blog: Snail In A Turtleneck.

I’ll be speaking about MongoDB at GotoCph

2011-04-01 by mookid8000 Leave a comment

I started playing around with MongoDB about a year ago, and since then I have grown to like it so much that I am going to try to tell what I know at the first ever Goto Copenhagen.

The talk is scheduled on the “Cloud and NoSQL” track on Thursday at 14:05.

A link directly to the abstract can be found here.

(PS: This is not an April fool’s day prank… :))

How to set the current culture in NServiceBus

2011-03-21 by mookid8000 3 Comments

Today we were experiencing some weird behavior when running an integration test with DillPickle, where – apparently – values of doubles would lose their decimal point when they were transferred in messages from our test to an NServiceBus service.

Stopping the service and inspecting the message in the queue quickly revealed a message that looked somewhat like this:

<Messages>
    <SetCurrentValueMessage>
        <Value>13.56</Value>
    </SetCurrentValueMessage>
</Messages>

</SetCurrentValueMessage>

</Messages>

which is all fine and dandy.

Now, I’m used to being Danish, so I know that we’re somewhat deviant in regards to our decimal point – “,” – so we quickly diagnosed the problem: Our integration tests were running with the invariant culture, to allow us to parse Gherkin files in English and use “.” as the decimal point – but Windows and everything else was running with da-DK, so 13.56 would be improperly deserialized to the value 1356 when it reached our NServiceBus service.

Solution: Normalize the culture of all the processes of our system.

Our first attempt was to modify the culture in our endpoint configuration like so:

public class EndpointConfiguration : IConfigureThisEndpoint, AsA_Server
{
    public EndpointConfiguration()
    {
        // does NOT work!
        Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture;
        Thread.CurrentThread.CurrentUICulture = CultureInfo.InvariantCulture;
    }
}

public class EndpointConfiguration : IConfigureThisEndpoint, AsA_Server

{

public EndpointConfiguration()

{

// does NOT work!

Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture;

Thread.CurrentThread.CurrentUICulture = CultureInfo.InvariantCulture;

}

but obviously this did not work, because NServiceBus does not deserialize messages on this thread!

Our solution was to create a message module, which seems to get called before transport messages are deserialized, setting the culture in there – like so:

public class SetCurrentCultureMessageModule : IMessageModule
{
    public void HandleBeginMessage()
    {
        // works!
        Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture;
        Thread.CurrentThread.CurrentUICulture = CultureInfo.InvariantCulture;
    }

    public void HandleEndMessage()
    {
    }

    public void HandleError()
    {
    }
}

public class SetCurrentCultureMessageModule : IMessageModule

{

public void HandleBeginMessage()

{

// works!

Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture;

Thread.CurrentThread.CurrentUICulture = CultureInfo.InvariantCulture;

}

public void HandleEndMessage()

{

}

public void HandleError()

{

}

In the future I’ll make sure that the culture is explicitly set in all processes of systems I am building. It’s kind of scary that errors could happen where stuff like “debit account 100.00” could be mis-interpreted as “debit account 10000”!! 😮