Fun with NoRM 4

2010-07-26 by mookid8000 Leave a comment

This time, a short post on how to model inheritance, which (at least in a class-oriented programming language) is one of the foundations of object-oriented programming.

Let’s take an example with a person, who has a home address that can be either domestic or foreign. Consider this:

public class Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }

    public Address HomeAddress { get; set; }
}

public abstract class Address
{
    public abstract string FormatAddress(string separator);
}

public class DomesticAddress : Address
{
    public string Street { get; set; }
    public string Number { get; set; }
    public string PostalCode { get; set; }
    public string City { get; set; }
    // and 17 other fields here, according to whatever is standard in your country

    public override string FormatAddress(string separator)
    {
        return string.Join(separator, new[] { Street + " " + Number, PostalCode + " " + City });
    }
}

public class ForeignAddress : Adress
{
    public string[] AddressLines { get; set; }

    public override string FormatAddress(string separator)
    {
        return string.Join(separator, AddressLines);
    }
}

public class Person

{

public string FirstName { get; set; }

public string LastName { get; set; }

public Address HomeAddress { get; set; }

}

public abstract class Address

{

public abstract string FormatAddress(string separator);

}

public class DomesticAddress : Address

{

public string Street { get; set; }

public string Number { get; set; }

public string PostalCode { get; set; }

public string City { get; set; }

// and 17 other fields here, according to whatever is standard in your country

public override string FormatAddress(string separator)

{

return string.Join(separator, new[] { Street + " " + Number, PostalCode + " " + City });

}

public class ForeignAddress : Adress

{

public string[] AddressLines { get; set; }

public override string FormatAddress(string separator)

{

return string.Join(separator, AddressLines);

}

Now, when I create a Person with a DomesticAddress and save it to my local Mongo, it looks like this:

var people = mongo.GetCollection<Person>();
people.Insert(new Person
{ 
    FirstName = "Mogens Heller", 
    LastName = "Grabe",
    HomeAddress = new DomesticAddress 
    { 
        Street = "Torsmark", 
        Number = "4", 
        // etc
    }
});

var people = mongo.GetCollection<Person>();

people.Insert(new Person

{

FirstName = "Mogens Heller",

LastName = "Grabe",

HomeAddress = new DomesticAddress

{

Street = "Torsmark",

Number = "4",

// etc

}

});

which is all fine and dandy – and in the db:

> db.Person.findOne();
{
    "FirstName": "Mogens Heller",
    "LastName": "Grabe",
    "HomeAddress": {
        "Street": "Torsmark",
        "Number": "4",
        // etc...
    }
}

> db.Person.findOne();

{

"FirstName": "Mogens Heller",

"LastName": "Grabe",

"HomeAddress": {

"Street": "Torsmark",

"Number": "4",

// etc...

}

which looks pretty good as well. BUT when I try to load the person again by doing this:

var people = mongo.GetCollection<Person>();
var me = people.FindOne();

1 2	var people = mongo.GetCollection<Person>(); var me = people.FindOne();

I get BOOM!!: Norm.MongoException: Could not find the type to instantiate in the document, and Address is an interface or abstract type. Add a MongoDiscriminatedAttribute to the type or base type, or try to work with a concrete type next time.

Why of course! JSON (hence BSON) only specifies objects – even though we consider them to be logical instances of some class, they’ re actually not! – they’re just objects!

So, we need to help NoRM a little bit. Actually the exception message says it all: Add a MongoDiscriminatedAttribute to the abstract base class, like so:

[MongoDiscriminated]
public abstract class Address
{
    public abstract string FormatAddress(string separator);
}

[MongoDiscriminated]

public abstract class Address

{

public abstract string FormatAddress(string separator);

}

That was easy. Now, if I do a db.People.drop(), followed by my people.Insert(...)-code from before, I get this in the db:

> db.Person.findOne();
{
    "FirstName": "Mogens Heller",
    "LastName": "Grabe",
    "HomeAddress": {
        "__type": "MongoTest.DomesticAddress, MongoTest",
        "Street": "Torsmark",
        "Number": "4",
        // etc...
    }
}

> db.Person.findOne();

{

"FirstName": "Mogens Heller",

"LastName": "Grabe",

"HomeAddress": {

"__type": "MongoTest.DomesticAddress, MongoTest",

"Street": "Torsmark",

"Number": "4",

// etc...

}

See the __type field that NoRM added to the object? As you can see, it contains the assembly-qualified name of the concrete type that resulted in that particular object, allowing NoRM to deserialize properly when loading from the db.

Now, this actually makes working with inheritance hierarchies and specialization pretty easy – just add [MongoDiscriminated] to a base class, resulting in concrete type information being saved along with objects of any derived type.

Only thing that would be better is if NoRM would issue a warning or an exception when saving something that could not be properly deserialized – this way, one would not easily get away with saving stuff that could not (easily) be retrieved again.

Fun with NoRM 3

2010-07-22 by mookid8000 Leave a comment

Third post in “Fun With NoRM” will be about how “the dynamism” of JavaScript and JSON is bridged into the rigid and statically typed world of C#. The thing is, in principle there’s no way to be certain that a JSON object returned from MongoDB will actually fit into our static object model.

Consider a situation where, for some reason, some of our orders have a field, PlacedBy, containing the name of the person who placed the order. Let’s see how things will go when adding the field and then querying all orders:

> use dbname
> var order = db.Order.findOne();
> order.PlacedBy = "El Duderino";
> db.Order.save(order);

> use dbname

> var order = db.Order.findOne();

> order.PlacedBy = "El Duderino";

> db.Order.save(order);

var orders = mongo.GetCollection<Order>();

foreach(var order in orders.Find())
{
    Console.WriteLine("Order #{0}", order.Number);
}

var orders = mongo.GetCollection<Order>();

foreach(var order in orders.Find())

{

Console.WriteLine("Order #{0}", order.Number);

}

– and BOOM ! – Cannot deserialize!: Norm.MongoException: Deserialization failed: type MongoTest.Order does not have a property named PlacedBy

This is actually pretty good, because this way we will never accidentally load a document with un-deserializable properties and save it back, thus truncating the document. But how can we handle this?

Well, NoRM makes it pretty easy: Make your model class inherit Expando, thus effectively becoming a dictionary. E.g. like so:

public class Order : Expando
{
   // ...
}

public class Order : Expando

{

// ...

}

Now we can do this:

var orders = mongo.GetCollection<Order>();

foreach(var order in orders.Find())
{
    Console.WriteLine("Order #{0}", order.Number);

    if (order.AllProperties().Any())
    {
        var props = order.AllProperties().Select(p => string.Format("{0}: {1}", p.PropertyName, p.Value));
        Console.WriteLine("\t{0}", string.Join(", ", props.ToArray()));
    }
}

var orders = mongo.GetCollection<Order>();

foreach(var order in orders.Find())

{

Console.WriteLine("Order #{0}", order.Number);

if (order.AllProperties().Any())

{

var props = order.AllProperties().Select(p => string.Format("{0}: {1}", p.PropertyName, p.Value));

Console.WriteLine("\t{0}", string.Join(", ", props.ToArray()));

}

– which yields:

Order #1
    PlacedBy: El Duderino
Order #2
Order #3

Order #1

PlacedBy: El Duderino

Order #2

Order #3

when run with a small DB containing three orders. Nifty, huh?

If you’re sad that you’ve given up your single opportunity to inherit something by deriving from Expando, just go ahead and implement IExpando instead. Then you need to suply a few members, but you can just redirect to an internal Expando in your class.

Next up, a post on how to model inheritance hierarchies… one of my favorites! 🙂

Fun with NoRM 2

2010-07-19 by mookid8000 Leave a comment

This second post in “Fun With NoRM” will be about querying…

How to get everything

Querying collections can be done easily with the anonymous types of C# 3 – e.g. the Order collection from my previous post can be queried for all orders like so:

var orders = mongo.GetCollection<Order>();

var allOrders = orders.Find();

var orders = mongo.GetCollection<Order>();

var allOrders = orders.Find();

How to use criteria

If we’re looking for some particular order, we can query by field values like so:

var orderNumber2 = orders.Find(new { Number = 2 });

1	var orderNumber2 = orders.Find(new { Number = 2 });

or by using the query operators residing in the static class Q:

var ordersWithNumberGreaterThan2 = orders.Find(new { Number = Q.GreaterThan(2) });

1	var ordersWithNumberGreaterThan2 = orders.Find(new { Number = Q.GreaterThan(2) });

More advanced criteria

The query operators can even be combined by combining criteria like so:

var ordersWithNumberBetween5And10 = orders.Find(new { Number = Q.GreaterThan(5).And.LessThan(10) });

1	var ordersWithNumberBetween5And10 = orders.Find(new { Number = Q.GreaterThan(5).And.LessThan(10) });

Now, what about the nifty dot notation? This an example where C#’s capabilities don’t cut it anymore, as everything on the left side in an anonymous type need to be valid identifiers – so no dots in property names!

This is solved in NoRM by introducing Expando! (not to be confused with ExpandoObject of .NET 4, even though they have similarities…)

Expando is just a dictionary, so to query by the field of an embedded object, do it like so:

var q = new Expando();
q["Items.Name"] = "beer";
var ordersWithBeer = orders.Find(q);

var q = new Expando();

q["Items.Name"] = "beer";

var ordersWithBeer = orders.Find(q);

As you can see, querying with NoRM is pretty easy – and I think the NoRM guys have found a pretty decent workaround in the case of dot notation, where C#’s syntax could not be bent further.

Stay tuned for more posts…

Fun with NoRM 1

2010-07-16 by mookid8000 Leave a comment

My previous posts on MongoDB have been pretty un-.NETty, in that I have focused almost entirely on how to work the DB through its JavaScript API. To remedy that, I shall write a few short posts on how to get rolling with MongoDB using NoRM, the coolest C# driver for MongoDB at the moment.

First post will be on how to connect and shove data into MongoDB.

Short introduction to NoRM

NoRM is “No Object-Relational Mapping”. It’s a .NET-driver, that allows you to map objects and their fields and aggregated objects into documents. I like NoRM because it’s successfully preserved that low-friction MongoDB-feeling, bridging C#’s capabilities nicely to those of JavaScript in the best possible way, providing some extra C#-goodies along the way. Please read on, you’ll see…

Connect to MongoDB

Easy – can be done like so:

using(var mongo = Mongo.Create("mongodb://hostname/dbname"))
{
   // go crazy in here!!1
}

using(var mongo = Mongo.Create("mongodb://hostname/dbname"))

{

// go crazy in here!!1

}

Inserting a few documents

Inserting documents with NoRM is easy – just create a class with fields and aggregated objects, and make sure the class either has a property named something like “Id” or has a property decorated with [MongoIdentifier], e.g. like so:

public class Order
{
    public Order()
    {
        Id = ObjectId.NewObjectID();
        Items = new List<Item>();
    }

    public ObjectId Id { get; set; }
    public int Number { get; set; }
    public List<Item> Items { get; set; }
}

public class Item
{
    public string Name { get; set; }
    public int Amount { get; set; }
}

public class Order

{

public Order()

{

Id = ObjectId.NewObjectID();

Items = new List<Item>();

}

public ObjectId Id { get; set; }

public int Number { get; set; }

public List<Item> Items { get; set; }

}

public class Item

{

public string Name { get; set; }

public int Amount { get; set; }

}

– and then go ahead and pull a strongly typed collection and insert documents into it:

var orders = mongo.GetCollection<Order>();

orders.Insert(new Order {
    Number = 1,
    Items = {
        new Item { Name = "beer" },
        new Item { Name = "nuts" },
    }
});

var orders = mongo.GetCollection<Order>();

orders.Insert(new Order {

Number = 1,

Items = {

new Item { Name = "beer" },

new Item { Name = "nuts" },

}

});

Now, to make this work I need to create five 200-line XML-files with mapping info etc. </kidding> no, seriously – that’s all it takes to persist an entire aggregate root!!

Pretty cool, eh? That’ s what I meant when I said low friction. Stay tuned for more posts, e.g. on how to query a collection…

I will be speaking about NoSQL and MongoDB

2010-06-13 by mookid8000 1 Comment

as seen from the eyes of a .NET developer at two events in June (in Danish).

The first event is a JAOO Geek Night at Dong Energy in Skærbæk on Tuesday June 29th at 4:30 pm. You can read more about the free JAOO Geek Nights here.

The other event is a meeting in Århus .NET User Group, which is the day after, on Wednesday June 30th at 6 pm – you can sign up via Facebook ~~here~~.

I’m really looking forward to it, because I think we will have some interesting discussions. And perhaps we can widen a few people’s horizons 🙂

Hope to see a lot of engaged people at both events.

Even more checking out MongoDB: The coolness continues

2010-03-31 by mookid8000 Leave a comment

One thing I started to think about after having looked at MongoDB was how to model things that are somehow connected – without the use of foreign keys and the ability to join stuff.

When working with documents, you generally just embed the data where it belongs. But what if I have the following documents:

"songs" collection:
{
    "name": "Factory",
    "artist": "Martha Wainwright"
}

{
    "name": "Winter At The Hamptons",
    "artist": "Josh Rouse"
}

"users" collection:
{
    "username": "mookid",
    "name": "Mogens Heller Grabe"
}

{
    "username": "duderino",
    "name": "The Dude"
}

"songs" collection:

{

"name": "Factory",

"artist": "Martha Wainwright"

}

{

"name": "Winter At The Hamptons",

"artist": "Josh Rouse"

}

"users" collection:

{

"username": "mookid",

"name": "Mogens Heller Grabe"

}

{

"username": "duderino",

"name": "The Dude"

}

– and I want to constrain access to the songs, allowing me to see both songs, and The Dude to see only Factory?

My first take was to simply add username in an array inside each song, like so:

"songs" collection:
{
    "name": "Factory",
    "artist": "Martha Wainwright",
    "allowed": ["mookid", "duderino"]
}

{
    "name": "Winter At The Hamptons",
    "artist": "Josh Rouse",
    "allowed": ["mookid"]
}

"songs" collection:

{

"name": "Factory",

"artist": "Martha Wainwright",

"allowed": ["mookid", "duderino"]

}

{

"name": "Winter At The Hamptons",

"artist": "Josh Rouse",

"allowed": ["mookid"]

}

– and this will work well with indexing, which can be done like this:

db.songs.ensureIndex({"allowed": 1})
db.songs.find({"allowed": "mookid"})  // will use the index :)

1 2	db.songs.ensureIndex({"allowed": 1}) db.songs.find({"allowed": "mookid"}) // will use the index :)

But here comes the problem: What if each song should be displayed along with the name of who can access that particular song? I need to embed more stuff in the array, e.g. like so:

"songs" collection:
{
    "name": "Factory",
    "artist": "Martha Wainwright",
    "allowed": [{
        "username": "mookid", 
        "name": "Mogens Heller Grabe"
    }, {
        "username": "duderino", 
        "name": "The Dude"
    }]
}

{
    "name": "Winter At The Hamptons",
    "artist": "Josh Rouse",
    "allowed": [{
        "username": "mookid", 
        "name": "Mogens Heller Grabe"
    }]
}

"songs" collection:

{

"name": "Factory",

"artist": "Martha Wainwright",

"allowed": [{

"username": "mookid",

"name": "Mogens Heller Grabe"

}, {

"username": "duderino",

"name": "The Dude"

}]

}

{

"name": "Winter At The Hamptons",

"artist": "Josh Rouse",

"allowed": [{

"username": "mookid",

"name": "Mogens Heller Grabe"

}]

}

There’s a challenge now in keeping this extra “denormalized” piece of information up-to-date in case a user changes his name, etc. – but let’s just assume we’ve handled that.

Now here comes the cool thing: It’s cool that MongoDB can index “into” an array, but it can actually index “into” anything! Just tell it where to go, using the Dot Notation.

That means I can perform the same search as I did above like so:

db.songs.ensureIndex({"allowed.username": 1})
db.songs.find({"allowed.username": "mookid"})  // will use the index :)

1 2	db.songs.ensureIndex({"allowed.username": 1}) db.songs.find({"allowed.username": "mookid"}) // will use the index :)

How cool is that?? (pretty cool, actually!)

More checking out MongoDB: Updating

2010-03-19 by mookid8000 Leave a comment

In MongoDB, there’s no way to lock a database, collection, or document. The ability to work without locking is a requirement for any db that wishes to be horizontally scalable, and obviously this imposes some limitations and/or possibilities (depending on your point of view :)).

If you want all the goodness that document-orientation brings, it seems we need to cope with this non-locking database.

So how DO you update stuff in MongoDB? And, more importantly: how do you update stuff without race conditions?

In one of my previous posts on MongoDB, I mentioned that the unit of atomicity is a document – i.e., either a document gets saved/updated/deleted or it doesn’t. That must mean that we can count on updating one document only (or not), so we should build our applications so they can work without requiring multiple documents to be updated to be consistent ([1. Which is good practice anyway! In my experience, long and wide db transactions are often used, not to enforce a strict consistency as much as to allow scenarios like: “when this happens, this should also happen”. But that kind of logic can often be handled by something else, e.g. by a publishing events reliably to other processes (logically and/or physically), that handles the side-effects.]).

First, let’s take a look at how to actually update a document.

Naïve attempt to update a document

Well, we could do this:

> use myblog
switched to db myblog
> var doc = {'headline': 'Just checking', 'tags': ['nifty']}
> db.posts.save(doc)
> // omgwtfbbq, we forgot to tag with 'test' as well... let's correct it:
> doc.tags[1] = 'test'
test
> // let's go get a cup of coffee...
> // ....
> // - aand now we're back - let's hit save
> db.posts.save(doc)
> db.posts.find()
{ "_id" : ObjectId("4b965229bf4a0000000043bc"), "headline" : "Just checking", "tags" : [ "nifty", "test" ] }

> use myblog

switched to db myblog

> var doc = {'headline': 'Just checking', 'tags': ['nifty']}

> db.posts.save(doc)

> // omgwtfbbq, we forgot to tag with 'test' as well... let's correct it:

> doc.tags[1] = 'test'

test

> // let's go get a cup of coffee...

> // ....

> // - aand now we're back - let's hit save

> db.posts.save(doc)

> db.posts.find()

{ "_id" : ObjectId("4b965229bf4a0000000043bc"), "headline" : "Just checking", "tags" : [ "nifty", "test" ] }

That is, if you go and save a document that already has an ID, any existing document with that ID will be updated.

This would work if we were the only client on the db. But what if someone was editing the post in that same moment, adding another tag as well? Well, if he was unfortunate enough to save his edits when we were out for coffee, his changes would be lost.

One way to actually do it

By using the update function!

update accepts the following four arguments:

criteria – document selector that specifies which document to be updated
objNew – document to save
upsert – bool to specify auto-insert if document does not exist (“update if present, insert if missing”)
multi – bool to allow updating multiple documents that match the criteria (default is only first document)

Actually, as you can now probably see, save(doc) is just a shorthand for update({}, doc, true, false) – an upsert with the document we’re saving.

This way, we could easily add an incrementing version field to our documents to make sure that the version we’re saving is the version we retrieved.

Let’s try it out:

> post = {'headline': 'has a version field', 'tags': ['nifty'], 'version': 1}
{
        "headline" : "has a version field",
        "tags" : [
                "nifty"
        ],
        "version" : 1
}
> db.posts.save(post)
> // now we're editing it
> post['tags'][1] = 'test'
test
> post['version']++
1
> post
{
        "headline" : "has a version field",
        "tags" : [
                "nifty",
                "test"
        ],
        "version" : 2,
        "_id" : ObjectId("4b9884d4b54e000000006c69")
}
> // now someone else retrieves the post
> someoneElsesPost = db.posts.findOne()
{
        "_id" : ObjectId("4b9884d4b54e000000006c69"),
        "headline" : "has a version field",
        "tags" : [
                "nifty"
        ],
        "version" : 1
}
> // and we save it, setting criteria to the version we retrieved
> db.posts.update({'version': 1}, post); db.$cmd.findOne({getlasterror: 1})
{ "err" : null, "updatedExisting" : true, "n" : 1, "ok" : 1 }
> // as you can probably tell, n:1 means that 1 document was updated...
> //
> // now that other guy makes an edit and tries to save it
> someoneElsesPost['tags'][1] = 'json'
json
> db.posts.update({'version': 1}, someoneElsesPost); db.$cmd.findOne({getlasterror: 1})
{ "err" : null, "updatedExisting" : false, "n" : 0, "ok" : 1 }
> // 0 documents were updated! Good!

> post = {'headline': 'has a version field', 'tags': ['nifty'], 'version': 1}

{

"headline" : "has a version field",

"tags" : [

"nifty"

"version" : 1

}

> db.posts.save(post)

> // now we're editing it

> post['tags'][1] = 'test'

test

> post['version']++

> post

{

"headline" : "has a version field",

"tags" : [

"nifty",

"test"

"version" : 2,

"_id" : ObjectId("4b9884d4b54e000000006c69")

}

> // now someone else retrieves the post

> someoneElsesPost = db.posts.findOne()

{

"_id" : ObjectId("4b9884d4b54e000000006c69"),

"headline" : "has a version field",

"tags" : [

"nifty"

"version" : 1

}

> // and we save it, setting criteria to the version we retrieved

> db.posts.update({'version': 1}, post); db.$cmd.findOne({getlasterror: 1})

{ "err" : null, "updatedExisting" : true, "n" : 1, "ok" : 1 }

> // as you can probably tell, n:1 means that 1 document was updated...

> //

> // now that other guy makes an edit and tries to save it

> someoneElsesPost['tags'][1] = 'json'

json

> db.posts.update({'version': 1}, someoneElsesPost); db.$cmd.findOne({getlasterror: 1})

{ "err" : null, "updatedExisting" : false, "n" : 0, "ok" : 1 }

> // 0 documents were updated! Good!

– good thing he didn’t get through with that one! 🙂

As you can see, we can easily implement optimistic concurrency on each document by constraining updates to the version we checked out. But, as I will show next, you can actually do a lot of things on the server.

Server-side document updates

Instead of retrieving an entire document, modifying it, and saving it back again with the risk of overwriting someone else’s edits, we can ask the server to make edits as smaller operations. E.g. our attempt to add a missing ‘test’ tag to the document could have been done like this:

> db.posts.update({'_id': ObjectId("4b9884d4b54e000000006c69")}, { $push: { 'tags': 'test' } });
> db.posts.find()
{
        "_id" : ObjectId("4b9884d4b54e000000006c69"),
        "headline" : "has a version field",
        "version" : 2,
        "tags" : [ "nifty", "json", "test" ]
}

> db.posts.update({'_id': ObjectId("4b9884d4b54e000000006c69")}, { $push: { 'tags': 'test' } });

> db.posts.find()

{

"_id" : ObjectId("4b9884d4b54e000000006c69"),

"headline" : "has a version field",

"version" : 2,

"tags" : [ "nifty", "json", "test" ]

}

See how the code $push modifier was used to push a value into the array… this stuff is great. But here, we have a race condition again – what if someone added the ‘test’ tag almost the same time as we did? Then two ‘test’ tags would be present in the array.

One way is to constrain the update by id and the absence of ‘test’ in the tags array:

> db.posts.update({
    '_id': ObjectId("4b9884d4b54e000000006c69"), 
    $neq { 'tags': 'test'} 
}, { 
    $push: { 'tags': 'test' } 
});

> db.posts.update({

'_id': ObjectId("4b9884d4b54e000000006c69"),

$neq { 'tags': 'test'}

}, {

$push: { 'tags': 'test' }

});

– another is to use the $addToSet function, which makes MongoDB treat the array as a set:

> db.posts.update({'_id': ObjectId("4b9884d4b54e000000006c69") }, { $addToSet: { 'tags': 'test' } });

1	> db.posts.update({'_id': ObjectId("4b9884d4b54e000000006c69") }, { $addToSet: { 'tags': 'test' } });

Nifty!

My conclusion (so far) is that an application can get a huge benefit from using the various modifier operations – performance-wise (obviously), but probably also UX-wise as well… It’s a step in another direction from the usual CRUD scenarios that I usually compulsively associate with the word “update”, and I imagine it could be made to reflect the user’s interactions with the system.

I am thinking that the majority of the user’s interactions with the system could (and probably should) be put on the form

db.collection.update({
        [assumed state before]
}, {
        [modifier operations to "migrate" one or
         more documents to the new state]
}, upsert, multi); db.$cmd.findOne({getlasterror: 1})

db.collection.update({

[assumed state before]

}, {

[modifier operations to "migrate" one or

How to perform updates across multiple documents

My first thought is that this situation should be avoided when working with a document-oriented db. I think most people will agree with this one.

I am pretty unsure of this, actually… The rest of this post is just a few thoughts on my first take on this, should I need to do this. Comments are greatly appreciated!

The problem in updating multiple documents is that we can perform an update on one document at a time, each time checking if the update went well or not. But there’s no way to (consistently) roll back update #1 if update #2 fails. So this means that there’s only one way: Forward! But how to proceed then, when an update failed?

How do we usually do stuff reliably across boundaries of multiple things that may or may not succeed, allowing us to handle errors as gracefully as possible and proceed thereafter?

I’m thinking that asynchronous reliable one-way messaging is the answer to this.

So if an application ever needs to update multiple documents in the most reliable way possible, it should probably perform one document update per “transaction” – in NServiceBus terminology that would be updating one document per message handler. And then the handler should throw an exception if an update unexpectedly fails.

But again: I’m thinking that this situation should be avoided at all costs with a document-oriented db. If ACID is required, the application should probably have a RDBMS on the side, or implement some kind of transactional mechanism in the document store.

Conclusion

That concludes my little learning series of MongoDB posts.

I must say that I am intrigued by all the NoSQL discussions currently going on in the communities, and I think it is always a sign of health that we question the technologies we use.

I am entirely convinced that document dbs could and should have been used for some parts of systems that I have experienced, and I am blown away by the lack of friction when starting up a project on top of a schemaless db.

As a .NET dude, I am convinced that the future will see more .NET systems built with more than one db underneath – e.g. with MongoDB for all the “soft parts” of the system, NHibernate on SQL Server for the few things that by nature require ACID, and then some NHibernate Search/Lucene/Solr.NET for full-text indexing and searching capabilities etc.

More checking out MongoDB: Querying

2010-03-07 by mookid8000 Leave a comment

In my first post about MongoDB, I touched querying very lightly. Querying is of course pretty important to most systems, so it’s fair to dedicate a separate post to the subject.

Querying in MongoDB works by sending a document to the server, e.g. in the following snippet I create a document with a post ID

var somePost = db.posts.findOne({"_id": ObjectId("00112233445566778899aabb")})

1	var somePost = db.posts.findOne({"_id": ObjectId("00112233445566778899aabb")})

– which can actually be even shorter, as the find and findOne functions accept an ObjectId directly as their argument, like so:

var somePost = db.posts.findOne(ObjectId("00112233445566778899aabb"))

1	var somePost = db.posts.findOne(ObjectId("00112233445566778899aabb"))

But how can I find a post with a specific slug? Easy, like so:

var somePost = db.posts.find({slug: "this-slug-probably-comes-from-a-url"})

1	var somePost = db.posts.find({slug: "this-slug-probably-comes-from-a-url"})

But how does this perform? It’s easy to examine how queries are executed with the explain() function, like so:

> db.posts.find({slug: "this-slug-probably-comes-from-a-url"}).explain()
{
        "cursor" : "BasicCursor",
        "startKey" : {

        },
        "endKey" : {

        },
        "nscanned" : 10000,
        "n" : 1,
        "millis" : 11,
        "allPlans" : [
                {
                        "cursor" : "BasicCursor",
                        "startKey" : {

                        },
                        "endKey" : {

                        }
                }
        ]
}

> db.posts.find({slug: "this-slug-probably-comes-from-a-url"}).explain()

{

"cursor" : "BasicCursor",

"startKey" : {

"endKey" : {

"nscanned" : 10000,

"n" : 1,

"millis" : 11,

"allPlans" : [

{

"cursor" : "BasicCursor",

"startKey" : {

"endKey" : {

}

]

}

– yielding some info about the execution of the query. I don’t know exactly how to interpret all this, but I think I get that "nscanned": 10000 means 10000 documents were scanned – and in a collection with 10000 documents, that’s not entirely optimal as it implies a full table scan. Now, let’s make sure that our query will execute as fast as possible by creating an index on the field ( _id is always automatically indexed):

db.posts.ensureIndex({slug:1})

1	db.posts.ensureIndex({slug:1})

Now lets explain() again:

> db.posts.find({slug: "post-no-454"}).explain()
{
        "cursor" : "BtreeCursor slug_1",
        "startKey" : {
                "slug" : "post-no-454"
        },
        "endKey" : {
                "slug" : "post-no-454"
        },
        "nscanned" : 1,
        "n" : 1,
        "millis" : 0,
        "allPlans" : [
                {
                        "cursor" : "BtreeCursor slug_1",
                        "startKey" : {
                                "slug" : "post-no-454"
                        },
                        "endKey" : {
                                "slug" : "post-no-454"
                        }
                }
        ]
}

> db.posts.find({slug: "post-no-454"}).explain()

{

"cursor" : "BtreeCursor slug_1",

"startKey" : {

"slug" : "post-no-454"

"endKey" : {

"slug" : "post-no-454"

"nscanned" : 1,

"n" : 1,

"millis" : 0,

"allPlans" : [

{

"cursor" : "BtreeCursor slug_1",

"startKey" : {

"slug" : "post-no-454"

"endKey" : {

"slug" : "post-no-454"

}

]

}

Wow! That’s what I call an improvement!

What about posts with a specific tag? First I tried the following snippet, because I learned that the special $where field could be put in a query document to evaluate a predicate server-side:

var niftyPosts = db.posts.find({$where: function() { return this.tags != null && this.tags.indexOf("nifty") != -1; }})

1	var niftyPosts = db.posts.find({$where: function() { return this.tags != null && this.tags.indexOf("nifty") != -1; }})

– and this actually works. This syntax is sort of clunky though. Luckily, MongoDB provides a nifty mechanism for arrays that automagically checks if something is contained in it. So my query can be rewritten to this:

var niftyPosts = db.posts.find({tags: 'nifty'})

1	var niftyPosts = db.posts.find({tags: 'nifty'})

Nifty!

Now, to take advantage of indexes, the special query document fields should be used. I showed $where above, but there are more – to name a few: $gt (greater than), $gte (greater than or equal), $lt (less than), $lte (less than or equal), $ne (not equal), and many more. For example, to count the number of non-nifty posts in February 2010:

> db.posts.find({
    date: {$gte: 20100201, $lt: 20100301},
    tags: {$ne: 'nifty'}
}).count()
0

> db.posts.find({

date: {$gte: 20100201, $lt: 20100301},

tags: {$ne: 'nifty'}

}).count()

Check out the manual for some great documentation on available operators.

Conclusion

Querying with MongoDB actually seems pretty cool and flexible. I like the idea that it’s possible to execute ad-hoc queries, and for most usage I think the supplied operators are adequate. The ability to supply a predicate function via $where seems really cool, but it should probably only be used in conjunction with one or more of the other operators to avoid a full table scan.

More checking out MongoDB: References

2010-03-05 by mookid8000 1 Comment

This post will touch a little bit on the mechanism used for references, and then a few thoughts on how document-orientation relates to OO.

Now – if you, like me, are into OO and normalized object models – the weirdness begins….. or maybe not?! (actually, I am not sure yet :))

In an OO world (and in a normalized RDB world as well), you reference stuff, thus reducing the amount of redundant information as much as possible. E.g. the names of countries should not be put in a column in your address table, each country should have a row in the countries table, and then be referenced by a countryId in the address table.

In a document-oriented world, you generally embed objects instead of referencing them. This is done for performance reasons, and because there’s no way to join stuff – which means that a stored ID/foreign key merely remains free for the client to manually use in additional queries.

When you do need to actually reference another document, use DBRef to create a reference, supplying the collection and the ID as arguments. E.g. like so:

> var author = {name: 'joe'}
> db.authors.save(author)
> author
{ "name" : "joe", "_id" : ObjectId("4b8ed200384e0000000065d8") }
> var post = {headline: 'hello there', author: new DBRef('authors', author._id)}
> db.posts.save(post)
> post
{
        "headline" : "hello there",
        "author" : {
                "$ref" : "authors",
                "$id" : ObjectId("4b8ed200384e0000000065d8")
        },
        "_id" : ObjectId("4b8ed4c8384e0000000065d9")
}

> var author = {name: 'joe'}

> db.authors.save(author)

> author

{ "name" : "joe", "_id" : ObjectId("4b8ed200384e0000000065d8") }

> var post = {headline: 'hello there', author: new DBRef('authors', author._id)}

> db.posts.save(post)

> post

{

"headline" : "hello there",

"author" : {

"$ref" : "authors",

"$id" : ObjectId("4b8ed200384e0000000065d8")

"_id" : ObjectId("4b8ed4c8384e0000000065d9")

}

This way, the reference is represented in a consistent manner which may – or may not – be picked up by the driver you are using. The C# driver can create DBRefs and follow them, but you don’t get to join stuff – you still need an extra query.

Embedding objects may seem a little clunky at first, but actually this plays nicely with some common OO concepts – take aggregation, for example: a blog post has an array of comments, each of which makes no sense without an aggregating post – i.e. comments live and die with their post. That’s an obvious sign that comments should be embedded in the post. So, instead of:

posts:
{'_id': ObjectId('11223344556677889900aabb'), headline: 'A post'}

comments:
{'_id': ObjectId('bbaa00112233554477669988'), text: 'hello there', post: { "$ref": "posts", "$id": ObjectId('11223344556677889900aabb') }}
{'_id': ObjectId('881122335544776699aa00bb'), text: 'hello again', post: { "$ref": "posts", "$id": ObjectId('11223344556677889900aabb') }}

posts:

{'_id': ObjectId('11223344556677889900aabb'), headline: 'A post'}

comments:

{'_id': ObjectId('bbaa00112233554477669988'), text: 'hello there', post: { "$ref": "posts", "$id": ObjectId('11223344556677889900aabb') }}

{'_id': ObjectId('881122335544776699aa00bb'), text: 'hello again', post: { "$ref": "posts", "$id": ObjectId('11223344556677889900aabb') }}

– you should do this:

{
    '_id': ObjectId('11223344556677889900aabb'), 
    headline: 'A post',
    comments: [
        {text: 'hello there'},
        {text: 'hello again'}
    ]
}

{

'_id': ObjectId('11223344556677889900aabb'),

headline: 'A post',

comments: [

{text: 'hello there'},

{text: 'hello again'}

]

}

Actually this makes me think about the concept of an aggregate root in DDD: an aggregate root “owns” the data beneath it, and is responsible for maintaining its own integrity. If you were to delete an aggregate root, all the data beneath it would dissappear.

This also fits kind of nicely with the fact that there’s no database transactions in MongoDB – i.e. there’s no way to issue multiple statements and have them rolled back in case of an error – there’s only documents, and either a document gets inserted/updated/deleted, or it doesn’t. So obviously, the document is the unit of atomicity, which fits (sort of nicely) with the aggregate root and its responsibility of keeping itself internally consistent.

Conclusion

The observations stated here pretty much make a document an aggregate root in the DDD sense – especially since only documents get an _id. There’s no obvious way to reference a particular comment inside the second post shown above.

If MongoDB’s performance is up to the task, data should probably be aggregated as much as possible into large documents. MongoDB’s limit is 4 MB per document, but I am unsure of how large documents should be before you should consider splitting them.

Maybe I am thinking too much about these things? Maybe I should just try and build something and see where my document modeling goes? Suggestions and comments are welcome 🙂

Checking out MongoDB

2010-03-03 by mookid8000 4 Comments

Having experienced a lot of pain using RDBMSs ([1. Usually because of abusing RDBMSs, actually. Storing an object model in a RDBMS is not painful as long as the tooling is right – e.g. by leveraging the amazing NHibernate. The pain comes when developers suddenly start implementing overly complex queries and doing reporting on top of a pretty entity model, modeling stuff OO style… ouch!]) as a default choice of persistence, having read a couple of blog posts about MongoDB, and being generally interested in widening my horizon, I decided to check out MongoDB.

This post is a write-as-I-go summary of the information I have gathered from the following places:

Getting MongoDB

Piece of cake! Download MongoDB from the download center and shove the binaries away somewhere on your machine. Default is for MongoDB to store its data in /data/db which translates to c:\data\db if you are using Windows – go ahead and create this directory. The MongoDB daemon can be started by running mongod.exe, which will accept connections on localhost:27017.

It will probably look something like the screenshot shown below.

An alternative data path can be specified on the command line, e.g. like so: mongod --dbpath c:\somewhere\else.

Accessing it with JavaScript

Run mongo.exe to start the Mongo Shell. It will probably look something like this:

In the MongoDB prompt, you can use JavaScript to access the db. Here’s a sample session of some commands I have found useful:

// lists the dbs in this mongo
> show dbs   
admin
local
>
> // change database context to some db ("myblog" - will automagically be created)
> use myblog   
switched to db myblog
>
> show collections
system.indexes
>
> // save a couple of documents in a collection named "posts"
> // (collection will be automagically created as well...)
> db.posts.save({
    headline: 'Notes to self about MongoDB', 
    slug: 'notes-to-self-about-mongodb', 
    tags: ['mongodb', 'nosql', 'nifty', 'c#']
 })
> db.posts.save({
    headline: 'Someday I want to check out CouchDB as well', 
    slug: 'someday-i-want-to-check-out-couchdb-as-well', 
    tags: ['couchdb', 'nosql', 'nifty', 'c#']
 })
>
> // show documents in a collection (returns a cursor, which will be iterated for the first 10 or 20 results - next pages can be retrieved with the 'it' command)
> db.posts.find()
{ "_id" : ObjectId("4b8e4281781b000000005cfc"), "headline" : "Notes to self about MongoDB", "slug" : "notes-to-self-about-mongodb", "tags" : [ "mongodb", "nosql", "nifty", "c#" ] }
{ "_id" : ObjectId("4b8e42cc781b000000005cfd"), "headline" : "Someday I want to check out CouchDB as well", "slug" : "someday-i-want-to-check-out-couchdb-as-well", "tags" : [ "couchdb", "nosql", "nifty", "c#" ] }

// lists the dbs in this mongo

> show dbs

admin

local

> // change database context to some db ("myblog" - will automagically be created)

> use myblog

switched to db myblog

> show collections

system.indexes

> // save a couple of documents in a collection named "posts"

> // (collection will be automagically created as well...)

> db.posts.save({

headline: 'Notes to self about MongoDB',

slug: 'notes-to-self-about-mongodb',

tags: ['mongodb', 'nosql', 'nifty', 'c#']

})

> db.posts.save({

headline: 'Someday I want to check out CouchDB as well',

slug: 'someday-i-want-to-check-out-couchdb-as-well',

tags: ['couchdb', 'nosql', 'nifty', 'c#']

})

> // show documents in a collection (returns a cursor, which will be iterated for the first 10 or 20 results - next pages can be retrieved with the 'it' command)

> db.posts.find()

{ "_id" : ObjectId("4b8e4281781b000000005cfc"), "headline" : "Notes to self about MongoDB", "slug" : "notes-to-self-about-mongodb", "tags" : [ "mongodb", "nosql", "nifty", "c#" ] }

{ "_id" : ObjectId("4b8e42cc781b000000005cfd"), "headline" : "Someday I want to check out CouchDB as well", "slug" : "someday-i-want-to-check-out-couchdb-as-well", "tags" : [ "couchdb", "nosql", "nifty", "c#" ] }

Now, I have successfully added two documents representing blog posts in a collection named posts. As you can see, MongoDB assigns some funky IDs to the documents.

> // lets get the first post ('find' and 'findOne' accept a query document as their first parameter)
> db.posts.findOne({'_id': ObjectId('4b8e4281781b000000005cfc')})
{
        "_id" : ObjectId("4b8e4281781b000000005cfc"),
        "headline" : "Notes to self about MongoDB",
        "slug" : "notes-to-self-about-mongodb",
        "tags" : [
                "mongodb",
                "nosql",
                "nifty",
                "c#"
        ]
}
>
> // now let's find all IDs ('find' and 'findOne' accept as their second parameter a document
> // specifying which fields to return)
> db.posts.find({}, {'_id': true})
{ "_id" : ObjectId("4b8e4281781b000000005cfc") }
{ "_id" : ObjectId("4b8e42cc781b000000005cfd") }
{ "_id" : ObjectId("4b8e4595781b000000005cfe") }
>

> // lets get the first post ('find' and 'findOne' accept a query document as their first parameter)

> db.posts.findOne({'_id': ObjectId('4b8e4281781b000000005cfc')})

{

"_id" : ObjectId("4b8e4281781b000000005cfc"),

"headline" : "Notes to self about MongoDB",

"slug" : "notes-to-self-about-mongodb",

"tags" : [

"mongodb",

"nosql",

"nifty",

"c#"

]

}

> // now let's find all IDs ('find' and 'findOne' accept as their second parameter a document

> // specifying which fields to return)

> db.posts.find({}, {'_id': true})

{ "_id" : ObjectId("4b8e4281781b000000005cfc") }

{ "_id" : ObjectId("4b8e42cc781b000000005cfd") }

{ "_id" : ObjectId("4b8e4595781b000000005cfe") }

That was a brief demonstration of the JavaScript API in the Mongo Shell. Now, let’s do this from C#.

Getting started with mongodb-csharp

Now, go to mongodb-csharp dowload section at GitHub and get a debug build of the driver. Create a C# project and reference the MongoDB.Driver assembly.

On my machine, punching in the following actually works:

[Test]
public void CanAddPost()
{
  using(var mongo = new Mongo())
  {
    mongo.Connect();

    var db = mongo["myblog"];
    var posts = db["posts"];

    posts.Insert(new Document
                   {
                     {"headline", "Post added from C#"},
                     {"slug", "post-added-from-csharp"},
                     {
                       "tags", new[]
                                 {
                                   "c#",
                                   "nifty"
                                 }
                       }
                   });
  }
}

[Test]

public void CanAddPost()

{

using(var mongo = new Mongo())

{

mongo.Connect();

var db = mongo["myblog"];

var posts = db["posts"];

posts.Insert(new Document

{

{"headline", "Post added from C#"},

{"slug", "post-added-from-csharp"},

{

"tags", new[]

{

"c#",

"nifty"

}

});

}

Now, I can verify that the document is actually in there by going back to the console and doing this:

> db.posts.find()
{ "_id" : ObjectId("4b8e4281781b000000005cfc"), "headline" : "Notes to self about MongoDB", "slug" : "notes-to-self-about-mongodb", "tags" : [ "mongodb", "nosql", "nifty", "c#" ] }
{ "_id" : ObjectId("4b8e42cc781b000000005cfd"), "headline" : "Someday I want to check out CouchDB as well", "slug" : "someday-i-want-to-check-out-couchdb-as-well", "tags" : [ "couchdb", "nosql", "nifty", "c#" ] }
{ "_id" : ObjectId("4b8e4b72091abb14e4000001"), "headline" : "Post added from C#", "slug" : "post-added-from-csharp", "tags" : [ "c#", "nifty" ] }
>

> db.posts.find()

{ "_id" : ObjectId("4b8e4281781b000000005cfc"), "headline" : "Notes to self about MongoDB", "slug" : "notes-to-self-about-mongodb", "tags" : [ "mongodb", "nosql", "nifty", "c#" ] }

{ "_id" : ObjectId("4b8e4b72091abb14e4000001"), "headline" : "Post added from C#", "slug" : "post-added-from-csharp", "tags" : [ "c#", "nifty" ] }

Nifty! Now lets show the posts from C#. On my machine the following snippet displays the headlines of all posts:

[Test]
public void CanShowPosts()
{
  using(var mongo = new Mongo())
  {
    mongo.Connect();

    var db = mongo["myblog"];
    var posts = db["posts"];

    Console.WriteLine("Posts");
    foreach(var post in posts.FindAll().Documents)
    {
      Console.WriteLine("    {0}", post["headline"]);
    }
  }
}

[Test]

public void CanShowPosts()

{

using(var mongo = new Mongo())

{

mongo.Connect();

var db = mongo["myblog"];

var posts = db["posts"];

Console.WriteLine("Posts");

foreach(var post in posts.FindAll().Documents)

{

Console.WriteLine(" {0}", post["headline"]);

}

– which is documented in the following screenshot:

Random nuggets of information

Document IDs

All MongoDB documents must have an ID in the _id field, either assigned by you (any object can be used), or automatically by MongoDB. IDs generated by MongoDB are virtually globally unique, as they consist of the following: 4 bytes of timestamp, 3 bytes of machine identification, 2 bytes of process identification, 3 bytes of something that gets incremented.

As a nifty consequence, the time of creation can be extracted from auto-generated IDs.

The ID type used by MongoDB can be created with ObjectId('00112233445566778899aabb') (where the input must be a string representing 12 bytes in HEX).

How are documents stored?

I you have not yet figured it out, documents are serialized to JSON – with the minor modification that it’s a BINARY version of JSON, hence it’s called BSON.

String encoding

UTF-8. No worries.

What about references?

I will research this and do a separate post on the subject. As MongoDB is non-relational, a “join” is – in principle – an unknown concept. There’s a mechanism, however, that allows for consistent representation of foreign keys that may/may not give you some extra functionality (depending on the driver you are using).

What about querying?

I will research this as well, posting as I go.

OR/M? (or OD/M?)

It is not yet clear to me how to handle Object-Document Mapping. Will require some research as well. As an OO dude, I am especially interested in finding out what a schema-less persistance mechanism will do to my design.

What else?

More topics include applying indices, deleting/updating, atomicity, and more. Implies additional blog posts.

Conclusion

My first impression of MongoDB is really good. It’s extremely easy to get going, and the few error messages I have received were easy to understand.

I am especially in awe with how little friction I encountered – mostly because of the schema-less nature, but also because everything just worked right away.