For now over three years we are working with unit tests. I’d like to share some of those experiences.
As we have two frameworks in place for our website, I can compare two different strategies for unit tests.
In our homemade legacy framework we were using SimpleTest. Mainly because I read PHP|Architect’s Guide to PHP Design Patterns and Simpletest was Jason‘s framework of choice.
We have a cron job running which runs all tests every hour. The results of the tests are being shown on a page. The results are also displayed in Nagios.
The problem with this setup is: If one test breaks it’s very easy to ignore it. Because most times tests fail there’s no big consequence. So the developer thinks: “I’ll look at that later”. And as human are naturally lazy usually “later” == “never”.
If it breaks, you cannot release
So in our new application framework, Trevi, we changed the setup: We switched to PHPUnit since Trevi sits on top of Zend Framework which has its own unit tests in PHPUnit.
Then, we took Cruise Control, hooked it up to Subversion and made it run our whole test suite on every commit. If one test breaks, a mail is sent to the project managers Leo and Maarten along with the information who made the tests break. Of course the developer wants to avoid the shame of being the test-breaker. So he will be very careful before committing.
There’s a second barrier to avoid broken tests: You cannot release code into production if the build breaks (= unit test fail).
So far, we have 343 unit tests. We don’t have 100% code coverage as some parts of the code are so simple it just does not make sense (e.g the views in the MVC framework). But we think we have dramatically reduced bugs by using them for our core components.
What’s your experience with unit tests? Do you consequently use them? What’s your code coverage?
We are using phpUnderControl, which is a really cool layer upon Cruisecontrol for specific PHP integration. So far I’m really happy with it. For code coverage: most of the time we aim 100%, which is not as hard as it sounds (http://flickr.com/photos/mastayoda/2476231742/).
I’m happy for you that you’ve now discovered PHPUnit which is a better unit testing framework in more ways than none… Simple Test appears to be, well… Simple?
Don’t get me wrong, it’s a well developed framework but it lacks a lot of features found in PHPUnit, and the PHPUnit community is more stable and growing at a pace quicker than that of Simple Test.
You’ll find that Simple Test as in regards to it’s loyal user base appear to be those from Sitepoint’s PAD forum for the most part, as from my experience most people not assoc. with that forum use PHPUnit by default.
Few people outside the regular posters of that forum haven’t even heard of Simple Test never mind used it… That alone speaks volumes in my opinion.
SimpleTest has one feature that PHPUnit is missing: The ability to crawl websites, submitting forms and clicking on links.
You may argue that this is not unit testing but functional/acceptance testing (an I should use Selenium for that). But you need functional testing too and Simpletest is a good tool for the job, as long as no JavaScript is involved.
I also prefer the Simpletest implementation of mocking to that of PHPUnit.