NSynthesis 0.1.1 has escaped!
- NUnit 2.5
- Rhino Mocks
- Nant
There are no new ideas, just old ones repackaged in new ways. Discuss.
Posted at
3:13 pm
1 comments
Labels:
.NET,
NSynthesis,
Synthesis,
TDD
Earlier this year George and I came up the with idea of trying to join up the mock interactions in our unit tests to improve the confidence that our tests were coherent and that all our mocked interactions hung together. The result of this was Synthesis.
The only problem was that I'm working on .NET projects at the moment. Enter NSynthesis.
The aim of NSynthesis is to provide 'code coverage for mocked interactions for .NET'. It uses PostSharp to detect mock expectations being set by your test code and verifies that there is a test of the real production implementation in the same test run. If this call is missing, we fail the build. Alex and Bernardo and I are busy working on getting NSynthesis to the point where we can have a 0.0.1 release and it is looking quite promising now. Out of curiosity, I thought I'd grab the trunk build and try to run NSynthesis on a subset of my current project.
I disabled the generic support as we are still working on this and it is not really stable yet. I also filtered the classes which we are using to only include the our services namespace. Services should be easy to unit test, as there is always a repository or another service between them and the real world. We also find this is where we have the most mocked interactions.
The results. 54 unit tests analysed, 12 unique mock calls made, 6 unique untested mock calls!
That's 50% of the mock interactions not being unit tested! Blimey!
So. There is definitely value in using this tool! It is not that this code is untested - there are lots of wired tests and acceptance tests. However, what it highlighted was that we could have tested a lot more code at a lower level, and possibly would have sped up our test suite as a result. It was a surprise for the team too - we all thought we were already unit testing everywhere we could.
When I extended NSynthesis to run against the whole code base, I ran into another problem. What broke NSynthesis was the situation where a class 'acquires' an implementation of an interface.
Here, my repository does not actually implement Exists itself, rather it derives from an existing implementation. In the code, this was our own BaseRepository, which in turn inherited from Castle's ARRepository<T>. Now when I mock the repository, I mock the interface. When I test my repository, I'm going to test the Repository class itself. NSynthesis then completely failed to realise that I have a tested implementation for IRepository because it is looking for a concrete implementation by a class which implements the interface IRepository. And BaseRepository does not.
I think getting this working could be fun :-)
Posted at
8:38 am
2
comments
Labels:
.NET,
NSynthesis,
Synthesis,
TDD
Following on from our presentation to EuRuKo 2008, George and I will be speaking at the next Thoughtworks Manchester Geek Night on the 8th April.
I'm an alumnus of UMIST - so I'm really looking forward to seeing the old place again, it has been a long time since I was last in Manchester!
Posted at
8:16 am
0
comments
Labels:
Ruby,
Synthesis,
TDD
George and I spoke to a bunch of friends about Synthesis a few weeks ago at the inaugural Reading Geek Night. Around six of us met in a pub in Reading and got really confused looks from the punters looking on on us as we waxed enthusiastically about TDD concepts and how we thought that they could be improved. I found it very interesting to get feedback from a group of really smart non ThoughtWorkers . They don't all practice “full on agile” but they certainly do understand what it means to build and ship working code, on time, with tight business pressures. It has taken a couple of weeks to digest the feedback, hence the slow time to post about the event.
So, here is another attempt to explain why we may need synthesis on our projects in the light of Reading Geek Night #1 and previous discussions with the guys at ThoughtWorks UK previous to this....
If you practice TDD, then the chances are that you already have a large number of unit tests. You may have a bunch of other automated tests of different types as well (functional, integration, performance...), and if you do, that's great. Keep doing that.
At the other end of the spectrum, you will also have some form of acceptance testing structure in place. The format of this varies from project to project. It could be a set of manual scripts which your Testing/QA team/Nominated Guy run, or it could be a bunch of system tests using FIT or one of the BDD frameworks. Maybe you paid for a commercial product and have something like a bunch of Rational Robot scripts.
We currently have a large disconnect between our high level tests and our low level unit tests, especially when you consider how confident you would be that the system works as desired by running one or more if the different types of test in the system.
At the bottom of the scale there is a system which has no tests. I would not be at all confident that this system worked. At the top end of the scale, I have a system which is running live in the real production environment and is being used by the actual user base. I am very confident that this system is working. At some point in the past, most businesses came to the conclusion that IT cannot blindly put systems live to discover if they work, hence all the interest in testing techniques.
High level system tests give us much more confidence that a system may work in production than unit tests because of the simple fact that they are exercising the production code in a more realistic manner; the data is more real, and all the interactions between the components are 'real', or as close to real as we can get in a test environment.
Unit tests provide a much weaker level of confidence due a number of issues when you want to use them to prove that a system works.
Unit tests are incomplete
We cannot unit test all of our code. Note that good design practices can massively reduce the amount of 'untestable' code in the system and by practicing TDD and running code coverage tools it is simple to provide a view of how well we are doing. However, there are always parts of the system which you either cannot test or neglect to unit test for some reason.
Unit tests are disconnected from other unit tests
Unit tests test a small amount of code, by definition, and often in isolation. If you are testing interactions between components then these are simulated using mocks. How can I be confident that I got my mock interactions correct? Also, how can I be confident that I have tested or even completed coding up the real version of whatever it is I am mocking? Again, we can mitigate this by reducing the number of interaction based unit tests and by writing state based unit tests when we can. However sometimes it is much more natural to follow the interaction test approach, and this incurs a risk that we are not simulating our interactions accurately.
Please don't take away from this that unit tests are bad. Unit tests are the lifeblood of a healthy project and are great for proving that the individual cogs are properly machined. What they do not tell you is whether you have the right number of cogs, or if they all fit together properly. If I want to know that components A,B, and C really play well together, then I need to write another aggregate test which puts ABC together and validates the unit tests got the interactions correct in a more realistic environment. George and I would call these functional tests, but you may use another term on your project. You may not have these tests on your project either – and that could be more worrying still – this means there could be gaps in your coverage at the levels that developers work at.
So, if I have functional tests plus unit tests, I am more confident that my system is working before I commit to running the slow system tests. B y running functional tests I have confidence that I have wired up all my well behaved units of code in a meaningful manner and that they are playing well with each other as I predicted. The chances are, that the wider system is going to work. I still am not as confident that the system works as I will be when my acceptance test suite is run by [insert machine/human here], but I can probably sleep well enough.
Now, some functional tests are inescapable and add a view into an aspect of the system that a unit test just cannot provide. A great example if you are using Spring would be to prove that you can get your components from the container and that they are wired up correctly. However, if you are purely proving that your simulated interactions are valid in the unit tests, then repetition is creeping into the process - which is wasteful.
This is where Synthesis comes in. Synthesis monitors the simulated interactions which you create in your unit tests and verifies that there is a corresponding unit test that exercises and validates the real object in your system. If everything joins up, then your build passes. If there are disconnects, then the build fails. So, if I have a unit tests for A, B, and C, then it is fine for me to simulate A's interactions with B and C using mocks, providing synthesis can match all my expectations with real tests which make the very same calls to the real A. The result is a synthetic bigger test, where A, B, and C are virtually linked by their expectations.
Obviously, there is a sliding scale of 'matching'. At one end you have basic method name matching, and at the other end of the scale there is full data matching for every call. The closer you get to true data matching, the more confidence is gained, but the more restrictions are placed on developers. We are not sure how far we need to go in order for a team to get an optimal balance between speed and safety – but I'd be very interested to get opinions on this matter :-)
Currently, Synthesis validates the call signature completely, but does not check the content of data passing between the mocks and data used to test an object for real. We plan to add this soon. However, there is a benefit to be had by ensuring that all your interactions join up. This will catch method mismatches, dynamic calls which are not tested such as Active Record queries, and gaps in your test coverage which have simply been missed by those fallible human developers!
Posted at
6:14 pm
3
comments
Labels:
Synthesis,
TDD
There are some common qualities that production and test code must possess in order to enhance their status as ‘good code’. Typical dimensions which one could apply may be conciseness, clarity of code, performance, elegance and extensibility (the list goes on). These are all good methods of assessing ‘good code’. However what differentiates good test code from good production code is not so clear when only these traditional qualities are taken into account. It may be more useful to consider what the drivers for the two type of code are.
For production code, these traditional values are useful, but cannot be assessed meaningfully if the code does not fulfil a business requirement. Ultimately this is the only driver of production code – it has to get the job done for the business. If your widget selling app for Widgets R Us cannot help to sell widgets, then no one will really care how speedy, elegant and extensible your Ultra Widget Framework is. The business code also needs to be robust enough for the business demands to allow the application to stay up and running cheaply enough for them to get a return on the build. It is a harsh world we live in and production code has to cut it in an unjust world.
So what about the test code? Test code has a different set of drives which compliment the production code. The primary driver is to assist in delivering high quality production code which robustly meets the requirements of our client. To this end the code also needs to drive the design and document the expected behaviour of our production code.
Driving the design of the code pushes the code towards delivery of production requirements, but this is only part of the story. Test code can used to accurately model the expected behaviour of the system and ensure that all key behaviour aspects of the system really do behave as per the requirements. By driving the design of the code with tests the system can meet the key delivery and quality drives of the code, but allows the developers the opportunity to make other improvements which improve the traditional values of the code. For example, a developer is more likely to refactor towards an elegant pattern which is emerging when a safety net in place than without one. If our tests measure performance then we are more likely to improve our performance qualities.
The test code is also a driver to document the behaviour accurately and clearly in order to provide a suitable level of traceability to the team and the stakeholders to demonstrate that the business need really have been met. When the test code is clear and comprehensive then the application benefits from not only the traditional test safety net, but also executable documentation that can highlight where and how the application has veered from the course of the business requirements.
So do traditional values not matter any more? Not at all - these values are crucial to quality code – in test and in production. However, you may assess the values differently in test and production code based when you consider what the code is driving to achieve. What may be reasonable in production code may be unsuitable for test code, as the application of a particular approach may obstruct the drives of test code to drive the design of the production code or clouds the documentation qualities of the test code by making a test less clear.
Put another way, just like in acting, it is important to consider your code’s motives first and then apply the traditional values in the context of what the code is trying to achieve.
Posted at
9:03 pm
0
comments
Labels:
TDD
We’ve all encountered it at some point or another. Your unit test is doing the job, but the test is so bogged down with setup code that it is impossible to see the wood for the trees. The interesting test code is in there somewhere, but where exactly?
The common causes of clogs in your test code are:
|
|
|
|
|
Posted at
5:04 pm
2
comments
Labels:
TDD
Remember that your test code is not production code. The purpose of your test code is to prove that your production code works and to provide live documentation. Qualities which make good test and production code are not necessarily the same. For example, it is important that a test case clearly demonstrates the expectations of a test case inline. This is especially true when working on an XP project, as these tests are your primary means of documenting the expected behaviour of your test classes. If another developer cannot read your tests easily and determine what was intended of the production code then trouble will ensue
Consider the following test case:
|
setMonitorExpectations method. More information is revealed:
|
buildAccountType.
|
|
|
Posted at
2:29 pm
2
comments
Labels:
TDD
George and I both consider ourselves fortunate enough to work as members of teams where TDD is practiced by default.
There are two reasons beyond the obvious as to why we find it relatively harder to code in a non TDD approach. Test Driving our code enhances our vision on how to design/model/instrument the application's universe. Also, starting with a test means work begins and ends with coding, not meetings, discussions or modelling our vision in pictures bound to be proven unrealistic when the first coding bottlenecks arise.
This is not to say that discussions or modelling are intrinsically bad things, a good brisk white board discussion can really help – but don't ever forget that this is a relatively abstract activity.
Tests have a much closer relationship with your code that allows you to discover and document the behaviour of your system in a much more detailed manner. In fact it is the close relationship between tests and code that can lead to problems. This relationship must be kept in balance if the process is to be successful.
Production code is the bread winner. This is where your business value lies. However, due to the nature of the TDD process, the production code is also somewhat dim-witted; it doesn't really have a clear vision or motivation in life. The test code is there to explain to the code what is expected of it and to highlight any mistakes the code makes by identifying in a precise manner why the code doesn't quite do the right thing and to point out how to head off in the correct direction. In some respects this could be likened to the relationship between a boxer and his trainer. Ultimately, it is the boxer who has to go into the ring and win the fight, but it is the trainer who gets the boxer into the zone mentally and physically.
Another equally important role of the test code is to document the expected behaviour of the production system. This is fantastic! Suddenly, the technical documentation is no longer a static and dusty tome on a shelf, rather an ever accurate and clear guide to what the code really does. If your code is not fit for purpose, this should be made glaringly obvious in your tests, and by changing your documentation you should be able to change the behaviour of your code to make it do the right thing in the right way.
However, we don't live in a perfect world and testing properly is not easy, despite the presence of opposable thumbs and shiny new Macs. Once you get past the simplistic examples and into the real world you find that it is difficult to write effective tests, and often the tests are much harder to craft than the actual code. In many ways, this should not come as a surprise. When you write a test, you are attempting to satisfy a number of aspects of the system: quality, design, documentation. And all of this expressed though the medium of a software language!
So, what makes our tests go bad? There seem to be a number of patterns starting to emerge, but to list them all here is going to take too long – I think George and I are going to have to break this out (hopefully with lots of assistance from our friends ;-)). In my mind the coarse categories of TDD anti patterns are:
Posted at
12:04 pm
2
comments
Labels:
TDD