How I test my apps...

Introduction:

After reading XP books like 3/4 years ago, I had the necessary information to get an insight on why tests were so important. I had a course in Uni where we went through most of the XP practices on a project, which I think was very important, but the concepts didn't sink in at that time.

About a year ago, I had the opportunity to start a project from scratch and that was great to put what I've read and tried to apply in other projects but only with a limited success because of the quality of the code base.

After reading the xUnit test patterns book, I got a zillion different concepts in my head. The book is really comprehensive. Still, it's not easy to get people to agree on terms like functional testing, accepting testing, end-to-end testing and so on... So, I'll use what I think it suits me, giving an explanation of what I mean by each.

Unit Tests - Tests that are focused on the smallest unit. In the case of java, it's a single class only. These tests are usually very comprehensive, they test pre and post conditions of methods, messages sent to collaborators and class invariants.

Integration tests - Tests that integrate my code with one or more external libraries and/or frameworks. This tests usually wire my application using spring/guice, do some database stuff, send mails or send messages through a messaging service like JMS and invoke web-services.

Acceptance tests - Tests that test the whole system from the user perspective. In a web application I'd use Selenium. I've also used Fitnesse to talk to a server through XMPP.

TDD:

TDD is a practice that I just love and can't really code without anymore. Even when I do a spike, it's so ingrained in myself that I have to start by writing a test. And maybe that's why I'm not so impressed with REPL consoles.

When I started doing some katas in scheme using the /The Little schemer/ book I had to create a small set of functions that enabled me to create tests with a name and a reporting utility to flag which tests failed.

REPL consoles are cool but they are just like manual testing, it doesn't scale and you can't re-use what you've done later.

I can now go back to the scheme code and know exactly what the function does just by looking at its tests. :)

TDD also helps me design my software using a push model. I create more functions/classes as I need instead of trying to guess what is necessary. I heard of people that are very good at building applications from the bottom up, I'm the opposite, I suck!

My style of TDD is very mock based, I also stub but only values not objects that I want to verify behavior.

My desires while testing:

Have good branch code coverage.

No duplication of tests, ie, minimise the number of tests that break for the same reason.

Use TDD to drive my design.

Fast build to have feedback as soon as possible.

So how do I test/design my applications:

I usually write ONE acceptance test per story.

If I'm using Fitnesse I will create the test remotely, disable it on the server, import it to my local server and re-enable it. I only re-enable it in the server after I've completed the story. Now, if I run this test it will fail. I think it's very important to keep verifying that your tests fail when they should fail, otherwise the test is useless or testing the wrong thing.

In order to carry on, I usually have to create an event that represents the user interaction. It can be just a wrapper for the HTTP parameters, an XMPP message, etc...

I then write ONE integration test for the component that will process the event. I create the expected response, I feed the event to the component and check that I don't get the expected response.

Next, I create ONE unit test for the component using the same event that I created for the integration test. This is usually the simplest happy path. I code the component until I get the expected response. Most of the times what happens is that in order to get the expected response I have to collaborate with other classes/libraries. I mock those and stub their responses.

Next, I go to the collaborators and write unit tests for those and code them using the same approach.

This will lead me down a path between objects until I hit a boundary or reach a dependency that already does what I need. So, when all of the mocked collaborators have been coded with the functionality I want, I'm done.

So far, I've been navigating down the path, it's time to go up and wire everything together using a DI library.

When done, I should be able to run the integration test and see it pass. If it doesn't, that means I have written tests that assumed something that was not true. I found that just by looking at the expected and the actual values of the integration test I can find straight way what the problem is.

With this I have my happy path implemented. :D

This is all good but I need to accommodate different scenarios, maintain class invariants, check pre and post conditions and handle errors.

By the way, If my system is only a single component then the acceptance test I wrote should pass now. If it’s not, I'll have to choose between coding a component at a time until it's completely done or code the happy paths for each of them till I get the acceptance test pass and come back later to each one for the rest of the coding. There are pros and cons for each approach and I don't have a preferred way yet...

So to finish off the components by tackling the different scenarios, handle errors, etc.., I use the following rule:

If a scenario/error doesn't need to integrate with an extra external library, i.e. you need to send an email when an error occurs but not when everything is ok. Than I just add unit tests to the appropriate places. Otherwise, I create an integration test and go from there all the way down as if it's a new feature.

Class invariants and pre and post conditions can usually be handled within unit tests only.

I continually do this until all the scenarios/errors/etc are covered and that means the story is done done. Normally when handling errors I try to find if the error can be handled then and there or if it has to be handled up the chain, which might mean I have to write tests to handle the error in another class.

How does this fulfills my desires:

Have good branch code coverage - The unit tests completely driven by TDD make sure I get all my code covered.

No duplication of tests - Because I've minimised the number of integration and acceptance tests, usually at most I get three tests broken when the happy path doesn't work.

Use TDD - By going layer by layer down the path I could TDD all my code in an organised way.

Fast build - The slow tests are the acceptance and integration test ones. By restricting those I speed up the build by a lot.

Do I get lots of bugs:

Exploratory testing is of course a very important part of the testing. I'm not going to say that no bugs were found, I would be lying, but all the bugs found so far were overlooked scenarios or just miss understandings of the requirements.

JB Rainsberger gave a talk about how integration tests are a scam. I agree with him so I used them only to make sure I test only the integration and not all the possible combinations. The same goes for acceptance tests. And if the client wants to write their own that’s ok but they are not going to run as part of the build.

What happens when I find a bug:

It depends on the bug. :P

If I can look at the logs and see in what object the problem is, I write a test with the appropriate context for the object and provide the input. If it fails, I'll fix the bug and check again through the process that found it, that it’s now fixed.

If I don't know the object but I know that the problem is in a given component, I'll write an integration test to find where the problem is. After I find the appropriate object I write a unit test and delete the integration test.

If I have no clue I write an acceptance test and go from there all the way down. But when I find the object I just write a unit test and delete all the other tests up.

I have found that with proper logging and keeping strict invariants in my classes I could find the errors straight away.

I just started reading Steve Freeman's and Nat Pryce's book and I found that they do some things that I also do, so I must have gotten something right. :)