Code coverage targets

The code at Salir.com has been developped under high schedule pressure. The team is already feeling swamped in their own "legacy code".

Based on my past experience, this would only get worse. If we just let things continue the same way, the throughput of the development team will get lower and lower, to a point where not even adding headcount will help much. So we have to act and reverse the situation. Not an easy task. Still, it's worth trying, because we all know how disastrous it can be for a company to throw the codebase away and start from scratch (ask Lotus, Netscape, or so many others).

I've never seen anyone succeed at rescuing a codebase which had become difficult to maintain. But Michel C. Feathers, in Working Effectively with Legacy Code, states that it is possible to reverse the software decay process. He says that the key to success is to put the code under test, and that it can be done given sufficient determination. So we're going to try.

We began a couple of sprints ago, and we reached 1.2% code (statement) coverage. It was a start. On the next sprint, we got to 3.8%. Nice! Now... is this good enough progress? Which is the target?

As a developer, I shouldn't care much. What's important is that there IS progress -- that is, that the codebase gets more maintainable rather than less maintainable over time. Also, quality is more important than quantity. In fact, you can even get to 100% coverage and have really bad tests: just don't write any assertions in your tests. Yes. I've seen teams doing that. As a manager, however, I want to know how much effort we're spending on what. First because I want to be able to change plans if we're not getting anywhere. Second because I must be able to justify the expense to my manager, the company's CEO.

Searching the internet I found the question of "how much coverage is good enough" several times, but no useful reasoned answers. Of course you can answer "only 100% coverage is good coverage" (has he ever tried to actually achieve it?), or "quantity is not important, quality is". These answers might be correct, but they are not useful.

So, searching for a useful answer, I've identified three large open-source projects written in our language (that's PHP, yuck!) which don't show serious maintainability problems. I've obtained measurements for them. The idea is that if we get as good as Zend Framework, Symfony, or Drupal, that's good enough for us.

The results:
Symfony: 75% statement (line?) coverage.
Zend Framework: 80% required, 90% encouraged (unclear whether it's statements or lines)
Drupal: 82% line coverage

So I'll set our target to 70% (which, coincidentally, is where PHPUnit paints its bars green) and see how we fare. At 1.5% each sprint, this will take 50 sprints, or roughly two years. That's OK.

Remember: the intent is to have a hint on progress, not to do Management By Objectives! As Alberto Savoia nicely states, Software Metrics Don't Kill Projects, Moronic Managers Kill Projects.

Nov 3rd update: in the last sprint we exceeded our code coverage targets, albeit by a small amout. Today during our sprint planning meeting, the developers were checking whether the modules to be changed already had unit tests. In surprisingly many cases they found they did, and this made them give lower estimates for those changes. I can draw two conclusions: (1) after 4 2-week sprints using unit tests, the team fully appreciates the value of having unit tests in place; (2) even though we didn't explicitly plan what to put under test first, the developers seem to have made the right choice and chosen those modules which were most likely to be changed again in the near future.