Unit test practice at tilllate.com

For now over three years we are working with unit tests. I’d like to share some of those experiences.

As we have two frameworks in place for our website, I can compare two different strategies for unit tests.

In our homemade legacy framework we were using SimpleTest. Mainly because I read PHP|Architect’s Guide to PHP Design Patterns and Simpletest was Jason‘s framework of choice.

We have a cron job running which runs all tests every hour. The results of the tests are being shown on a page. The results are also displayed in Nagios.

The problem with this setup is: If one test breaks it’s very easy to ignore it. Because most times tests fail there’s no big consequence. So the developer thinks: “I’ll look at that later”. And as human are naturally lazy usually “later” == “never”.

If it breaks, you cannot release

So in our new application framework, Trevi, we changed the setup: We switched to PHPUnit since Trevi sits on top of Zend Framework which has its own unit tests in PHPUnit.

Then, we took Cruise Control, hooked it up to Subversion and made it run our whole test suite on every commit. If one test breaks, a mail is sent to the project managers Leo and Maarten along with the information who made the tests break. Of course the developer wants to avoid the shame of being the test-breaker. So he will be very careful before committing.

There’s a second barrier to avoid broken tests: You cannot release code into production if the build breaks (= unit test fail).

So far, we have 343 unit tests. We don’t have 100% code coverage as some parts of the code are so simple it just does not make sense (e.g the views in the MVC framework). But we think we have dramatically reduced bugs by using them for our core components.

What’s your experience with unit tests? Do you consequently use them? What’s your code coverage?