Are you a good craftsman?

In a previous post I ranted about CTOs who allow their products to accumulate technical debt.

Over the past few months, however, I've come to realize that the fault does not lie only with management: the level of craftsmanship displayed by the average coder is dismal -- at least in the environment where I move (spain, startups, new media, PHP). Today I stumbled upon this video by Uncle Bob Martin which makes me realize that the same tragedy also happens elsewhere [well, I already knew, but Uncle Bob provides visual proof -- check it out].

We could go back and forth discussing who is to blame: the employers who don't know how to choose and train their personnel? The people who postulate themselves as programmers without even knowing what they don't know? Others who know what they don't know, but are too lazy or too stubborn to learn and apply it? Our colleges and universities, who don't train would-be programmers properly? But.. what's the point? What would I do with that blame?

Applying the "think globally, act locally" principle, I've decided to start with my direct environment, where I can make the most impact: my team and, first and foremost, myself.

But, what should I do?
 
Let's face reality: the craft of computer programming is less than 60 years old. If you compare it to pottery, we would be in 24940 BCE. New programming languages appear, expand, and are forgotten every [mercurian] year. Even programming paradigms cycle fast enough that it's hard to keep track of their names. It's not enough to become good once: I and my team-mates can only be good software craftsmen by actively keeping track of all these developments, learning how to apply them, and applying them wisely and courageously.

So I vow to:

1.- Spend a few hours every week reading blogs & books about the craft of computer programming, and learning to apply what I read.
2.- Make everything in my hands to make my team mates do the same (which should be a lot, as I am their functional manager).
3.- Have the courage of applying these new learnings at work whenever we firmly believe doing so is in the best interest of our employer.


I hope this will make me a good software craftsman.

And you? What are you doing to be a good craftswoman or craftsman?

Scrum.org: Religious Schism or business disagreement?

You may be aware of this posting by Ken Schwaber for details. I can't tell yet whether it's a case of Religious Schism, Founder's Syndrome, or just plain disagreement among so-far-successful business partners.

In any case, I found Ken's statement about the Scrum Guide not being updated by the Scrum Alliance intriguing, and I used vi and tkdiff to analyze the differences between the new Scrum Guide published in Scrum.org and the old one published in ScrumAlliance.org.

This is what I've found:

• The document is now co-authored by Ken Schwaber and Jeff Sutherland.
  
• The old version was dated in May 2009, the new one in November same year.
• The old document didn't carry any copyright notices. The new one does (reserving all rights to the authors).
  
• An historical introduction with acknowledgements has been added.
• The first section changes title from "Introduction to Scrum" to "Purpose".
• The three "legs" are now three "pillars".
• A couple of spelling mistakes are corrected, and another couple added. Ditto for some minor punctuation and grammar issues.
  
• Two phrases have been added, the first one very necessary (as there was no definition of "impediment" in the old document), the second one somewhat of a non-sequitur, IMO:
• [...] The ScrumMaster also helps the Scrum Team do its best in an organizational environment that may not yet be optimized for complex product development. When the ScrumMaster helps make these changes, this is called “removing impediments.” [..]
  
• [...] Release planning is entirely optional. If Scrum teams start work without the meeting, the absence of its artifacts will become apparent as an impediment that needs to be resolved. Work to resolve the impediment will become an item in the Product Backlog. [...]
  
• The definition of the Daily Scrum changes from "a 15-minute status meeting" to "a 15-minute inspect and adapt meeting" -- which anyone familiar with Scrum will agree is a good, necessary change.
• The last tip has been moved into a last section titled "Final thoughts".
  

All in all, I don't find enough ingredients for a Religious Schism,  or even for Founder's Syndrome. So, until I get more data, I'll operate under the assumption that it's just a dispute over intelectual property of a work which is generating material revenue.

[P.S.: thanks to  Ron Jeffries for pointing out that "Cisma" is not English. I found the right word in English is "schism" -- also a transliteration of the Greek source, σχισμα. Corrected]
[P.S.: thanks to Michael James for making me notice that the last tip was moved to a "Final thoughts" section. Updated.]

If you're the CTO in a newly-founded startup...

Short version:

If you're the CTO in a newly-founded software-based startup, please read about technical debt and know these estimates:

• If you don't code unit tests and hardly ever refactor, you're probably accruing technical debt as fast as 2, or even 3 man·days of debt per man·day of work.
• The interest rate on this kind of debt is easily in the range of 7% ... monthly!

Long version:

My last three jobs have had many aspects in common, including web applications with low maintainability, a.k.a. legacy code. I could even say that I make a living out of legacy code: it's not until those companies realize that their development is dismayingly slow that they get ready to shell out a salary like mine to improve the situation.

How is it possible that young companies like those have already managed to entangle themselves in legacy code? In the most extreme case, the product was less than one year old when I joined and it was already exhibiting serious maintainability problems.

A start-up environment is specially prone to generating legacy code: start-ups hire (or are founded by) programmers with little or no experience; there is a strong sense of urgency pervading the culture (or else they won't survive long enough to hire me, so I'll never know about them); and, even if they know they are incurring in technical debt, that's not seen as a big problem in comparison to the financial debt problem.

So, if you're the CTO/CEO in a newly-founded startup, you can choose to do the same: hire inexperienced programmers, put them under schedule pressure, and accumulate technical debt. That's a proven good path: those three companies I mentioned earlier have been successful enough by following it. Futhermore, it's a path which enlarges my job market, for which I will thank you profusely.

My recommendation, however, is to carefully avoid technical debt -- or at least the kinds Uncle Bob calls "mess", or Steve McConnell classifies as "first kind". Why? Because the interest on technical debt can easily run as high as 7% monthly. What would you tell your CFO if he used a credit line at 125% yearly interest?

(How did I get to this figure? I did a quick & dirty estimation from just a few personal observations: that writing new code without any attention to maintainability can easily be 3 times as fast as with; that after about 12 man·months of doing that, development slows to a crawl and the company has to hire more people just to keep moving. Keep the metaphor and do the math: roughly 7% a month will bring your 300% productivity down to 50% in 12 [man·]months.)


Doing without automated testing is one specific kind of technical debt which each of those three companies incurred into, and which is tremendously onerous to pay back -- so I'm warning you against it specifically: please make sure at least a few in your development team are familiar with automated testing, and that they are given enough slack -- specially in the first few weeks of the project -- to put this knowledge to use. You'll get a better quality product and you'll avoid finding yourself in a predicament just a few months down the road.

Certifications

I just received my CSP certification.

Looks like I'm thus the 3rd CSP in Spain (the other two currently listed in the Scrum Alliance directory are Andy P. Murthar and David Doctor Sánchez-Migallón).

I ask myself the same question I asked David Doctor a couple of months ago: Will this be of any use? I mean... beyond giving me the right to paste this here:

I went through the process because I dream of becoming an agile coach. I mean a real one: someone who helps teams in their path to agility, not just a CSC. And I have the (perhaps naive) belief that these certs & seals will eventually give me some marketing leverage in finding my guinea pigs.

Simple metrics for Scrum

After my talk yesterday at the Barcelona PHP Conference I received a lot of questions on the metrics we're using -- how we're building our burn-down charts, etc.

Here's what we're doing as of today. It's a mixture of ideas from "Scrum and XP from the Trenches", "Agile Estimation and Planning", and our own:

• The Product Backlog (PBL) is estimated in dimension-less Story Points via Planning Poker. For that, we run a planning poker session at least once a sprint. Note that we only use these for long-term planning.
  
• Before each Sprint Planning Meeting, we compute how many man·hours we will have available for the next sprint. We multiply by a focus factor (currently 55%). Until a couple of sprints ago, we used the focus factor obtained from the previous sprint, but that didn't work well and caused alternating sprints of too high/too low workload. So we decided to reset the value to sthg in the low-ish range and slowly bring it up as we feel confortable we can. I don't think we're done experimenting in this area.
  
• At each Sprint Planning Meeting, we (meaning Team+PO) choose a few items from the top of the PBL, break them down into tasks (in yellow post-its), and estimate each in hours ("ideal" hours, to be precise). We decide how many (and which) items to take based on these hours: the total must not exceed the product computed in the previous step. The post-its for the items taken go to the "Pendiente" (english: "pending") column on the whiteboard.
  
• The sum of the estimates for all the tasks taken into the sprint becomes the first point in the burn-down chart.
• The team updates the whiteboard daily by:
  
  • Writing two numbers into each task they've worked on: hours worked on it since the previous sprint & estimated hours remaining to completion.
    
  • Moving any completed tasks into the "completed" column.
  • Adding a blue post-it for each task that has been "willingly" taken into the sprint. Of course we only do this when we're clearly ahead of plan. ("Willingly" meaning we had the choice of taking it or not, in contrast with "forced" below).
    
  • Adding a red post-it (with an estimate) in the top row (labelled "Fuera de sprint", english: "out of sprint") for each unplanned task that has been forced into the sprint. We would prefer to avoid those, but the business requires that we take some of these -- they are mostly value-added tasks which just can't be planned because they depend on external events such as the sale of an advertising campaign. Nasty bugs are handled this way, too.
  • Tasks that should belong to the sprint but which we failed to identify during planning also get a red post-it, but that one goes in the row for the corresponding product item.
    
• I then add up the latest estimate available for each task on the "live" part of the whiteboard (that is: all except completed tasks) to obtain an estimate of work remaining. That is the value charted in the burn-down chart for the day. The manual addition is annoying, but it only takes me a couple of minutes -- which I happily pay in exchange for the many advantages of a real whiteboard.
• At the end of each sprint I produce a small set of statistics which I publish in the wiki page for that sprint (I also write the previous sprint's value by them to compare -- graphs might be better):
  
  INITIAL STATS (computed at the start of the sprint)
  
  A. Target velocity: sum of story points for all items initially taken into the sprint
  
  B. Estimated hours: sum of estimates in all yellow post-its at the start of the sprint -- as said, this is the first value plotted in the burn-down chart.
  C. Hours available: total man·hours available for the sprint work.
  D. Target focus factor (B/C): as a %
  
  FINAL STATS (computed only at the end)
  
  E. Real velocity: total story points actually completed in the sprint
  
  F. Real hours: actual man·hours available for the sprint work (i.e. C +/- sickness and alikes)
  G. Total estimate of completed planned tasks: sum of the initial estimates of the yellow and blue stickers which were actually completed. That is: B + blue stickers - descoped work.
  H. Final focus factor (G/F): as a %
  
  J. Hours charged to planned tasks: count of all hours written against all yellow and blue stickers completed in the sprint. This is the most painful to compute, but it's not more than two or three of minutes per sprint, so stop whining. Even my 9-year-old daughter can add that without a computer.
  K. Hours charged to unplanned tasks "inside the sprint"
  L. Estimation error ((J+K)/G-1): as a %
  M. Real focus factor ((J+K)/F): as %
  
  N. Hours charged to unplanned tasks "out of sprint": I often also decompose these in categories such as bugfixes, support, operations, advertising.
  
  P. Total hours charged (J+K+N)
  Q. Reporting error (1-P/F)

It's a lot of values, and the naming of the different "focus factors" can get confusing, but they allow us to tell, in the event of a planning problem, whether it was caused by excessive unplanned work, invalid estimation (missing tasks), or inaccurate estimation. The last two are just a sort of "checksum" so we can notice if we ever become lazy and stop reporting properly.

Charla en Barcelona PHP Conference 2009

Hoy he dado una charla en la Barcelona PHP Conference 2009 -- esencialmente sobre mi experiencia particular con Scrum y las lecciones que he ido sacando.

Podeis ver las transparencias (con sus notas, de las que he leído sin mucha vergüenza, pero aparentemente la audiencia no se ha dado cuenta) en SlideShare.

Code coverage targets

The code at Salir.com has been developped under high schedule pressure. The team is already feeling swamped in their own "legacy code".

Based on my past experience, this would only get worse. If we just let things continue the same way, the throughput of the development team will get lower and lower, to a point where not even adding headcount will help much. So we have to act and reverse the situation. Not an easy task. Still, it's worth trying, because we all know how disastrous it can be for a company to throw the codebase away and start from scratch (ask Lotus, Netscape, or so many others).

I've never seen anyone succeed at rescuing a codebase which had become difficult to maintain. But Michel C. Feathers, in Working Effectively with Legacy Code, states that it is possible to reverse the software decay process. He says that the key to success is to put the code under test, and that it can be done given sufficient determination. So we're going to try.

We began a couple of sprints ago, and we reached 1.2% code (statement) coverage. It was a start. On the next sprint, we got to 3.8%. Nice! Now... is this good enough progress? Which is the target?

As a developer, I shouldn't care much. What's important is that there IS progress -- that is, that the codebase gets more maintainable rather than less maintainable over time. Also, quality is more important than quantity. In fact, you can even get to 100% coverage and have really bad tests: just don't write any assertions in your tests. Yes. I've seen teams doing that. As a manager, however, I want to know how much effort we're spending on what. First because I want to be able to change plans if we're not getting anywhere. Second because I must be able to justify the expense to my manager, the company's CEO.

Searching the internet I found the question of "how much coverage is good enough" several times, but no useful reasoned answers. Of course you can answer "only 100% coverage is good coverage" (has he ever tried to actually achieve it?), or "quantity is not important, quality is". These answers might be correct, but they are not useful.

So, searching for a useful answer, I've identified three large open-source projects written in our language (that's PHP, yuck!) which don't show serious maintainability problems. I've obtained measurements for them. The idea is that if we get as good as Zend Framework, Symfony, or Drupal, that's good enough for us.

The results:
Symfony: 75% statement (line?) coverage.
Zend Framework: 80% required, 90% encouraged (unclear whether it's statements or lines)
Drupal: 82% line coverage

So I'll set our target to 70% (which, coincidentally, is where PHPUnit paints its bars green) and see how we fare. At 1.5% each sprint, this will take 50 sprints, or roughly two years. That's OK.

Remember: the intent is to have a hint on progress, not to do Management By Objectives! As Alberto Savoia nicely states, Software Metrics Don't Kill Projects, Moronic Managers Kill Projects.

Nov 3rd update: in the last sprint we exceeded our code coverage targets, albeit by a small amout. Today during our sprint planning meeting, the developers were checking whether the modules to be changed already had unit tests. In surprisingly many cases they found they did, and this made them give lower estimates for those changes. I can draw two conclusions: (1) after 4 2-week sprints using unit tests, the team fully appreciates the value of having unit tests in place; (2) even though we didn't explicitly plan what to put under test first, the developers seem to have made the right choice and chosen those modules which were most likely to be changed again in the near future.