Dogfooding NSynthesis

Earlier this year George and I came up the with idea of trying to join up the mock interactions in our unit tests to improve the confidence that our tests were coherent and that all our mocked interactions hung together. The result of this was Synthesis.

The only problem was that I'm working on .NET projects at the moment. Enter NSynthesis.

The aim of NSynthesis is to provide 'code coverage for mocked interactions for .NET'. It uses PostSharp to detect mock expectations being set by your test code and verifies that there is a test of the real production implementation in the same test run. If this call is missing, we fail the build. Alex and Bernardo and I are busy working on getting NSynthesis to the point where we can have a 0.0.1 release and it is looking quite promising now. Out of curiosity, I thought I'd grab the trunk build and try to run NSynthesis on a subset of my current project.

I disabled the generic support as we are still working on this and it is not really stable yet. I also filtered the classes which we are using to only include the our services namespace. Services should be easy to unit test, as there is always a repository or another service between them and the real world. We also find this is where we have the most mocked interactions.

The results. 54 unit tests analysed, 12 unique mock calls made, 6 unique untested mock calls!
That's 50% of the mock interactions not being unit tested! Blimey!

So. There is definitely value in using this tool! It is not that this code is untested - there are lots of wired tests and acceptance tests. However, what it highlighted was that we could have tested a lot more code at a lower level, and possibly would have sped up our test suite as a result. It was a surprise for the team too - we all thought we were already unit testing everywhere we could.

When I extended NSynthesis to run against the whole code base, I ran into another problem. What broke NSynthesis was the situation where a class 'acquires' an implementation of an interface.

Here, my repository does not actually implement Exists itself, rather it derives from an existing implementation. In the code, this was our own BaseRepository, which in turn inherited from Castle's ARRepository<T>. Now when I mock the repository, I mock the interface. When I test my repository, I'm going to test the Repository class itself. NSynthesis then completely failed to realise that I have a tested implementation for IRepository because it is looking for a concrete implementation by a class which implements the interface IRepository. And BaseRepository does not.

I think getting this working could be fun :-)