I stumbled across Agile Shrugged today and read a post on how the author experienced a lot of variation in their team velocity.   The author did not seem too concerned with all the “sawtooth” behavior of velocity chart because they felt the velocity graph was a “fairly accurate measure of our teams velocity”.  Let’s see if this qualitative assessment matches up with statistical process control, one of the Six Sigma tools.

In Six Sigma, a common tool Green Belts and Black Belts use to monitor their processes are control charts.  The control chart is an objective measurement tool to determine if the process under inspection can reliably produce what is expected of it in the future.  By examining the patterns of the data on the control chart we can understand if the process is “in control” or “out of control”.  If the process is “out of control”, then we need to inspect the process, discover the cause of the variation, “mod” the process and monitor the changes with…a control chart.

The diagram below is the velocity data from the team.  You can see the “sawtooth”, or oscillation, in velocity over the course of ten iterations.  The team does seem to have improved their velocity over time, but we still have an order of magnitude difference in the completed output of the team from week-to-week.  Is this significant?  Should we care about the oscillation?

To make a control chart, we need to run some basic statistics on the data: compute the mean, standard deviation (σ) and determine the upper and lower spec limits (mean ± 3σ).  Here are the results for this data set:

 mean: 6.6 ideal days    upper spec limit (USL): 24 ideal days
 σ:    5.8 ideal days    lower spec limit (LSL): -10.8 ideal days

Next, we will plot the data on a control chart showing the mean (green line), the USL (red line), 1σ and 2σ (purple lines).  I have eliminated the LSL because a negative velocity is meaningless.

Now here is the interesting part: looking at the patterns.  Using the Nelson Rules, we can examine the patterns in the control chart to see if the process is “in control” or “out of control”.  Here are the rules that I think are relevant for this chart (please review all the rules if you are interested in learning more):

1. One point is more than 3 standard deviations from the mean. Nope
2. Fourteen (or more) points in a row alternate in direction, increasing then decreasing. Only 8 points so far; maybe something to watch out for.
3. Four (or five) out of five points in a row are more than 1 standard deviation from the mean in the same direction.  Only 2 out of 5 data points.

What do we find by creating and examining the control chart?  Statistical confirmation that our qualitative opinion that our planning process is indeed fine.  Great!  Keep up the good work!

Dave Nicolette posted an interesting blog post about using function points (FP) as a means to compare productivity between IT projects using different processes.  I understand the concept – find a metric that speaks to the audience we are trying to communicate with that allows an apples-to-apples comparison.  However, why do I find this still an unsatisfactory answer to the productivity issue?

I tend to think this metric would only be useful for academic types or a research paper.  Sure, it is create more evidence that “Agile is great and waterfall is bad”, and how long do we have to wait for this evidence?  One business cycle?  Since I am on a roll smashing your good idea, I also think the barrier for entry is too high for people wishing to make the case Agile would be good for their organization.  Not only do they need to understand Agile practices & techniques, they now need to become FP counters.  Now we need to start counting FP on all our IT projects?  Is this something our industry has the skill to do?  The evidence I have seen shows we do not.

Yes, FP are “good” because ISO has made them the standard, there is clear way to count them (for the people who know) and academics (and attornies) love them.   There are real reasons why FP are not used in US business today.  I work with the ideal organization (large Fortune 500 company) for FP analysis, they are always hungry for metrics and they do not use FP.  I wonder why that is the case?  It is not due to a lack of education.  There are well-placed, knowledgable individuals aware of this metric, so there must be something else preventing FP.  We need to understand why people who know about this metric, can afford the consultants to do the analysis and still do not use the metric.  When we overcome those issues, then let’s talk about pushing this metric on Agile teams.  In the meantime, let’s stick with “basics” like throughput, takt time or use EVM (as much as I hate it – it is comprehensible to business folks).

One of the sessions I visited today was about what are the correct measurements to determine if your SOA initiative was successful.  One of the most interesting statistic from the speaker was this continuum of cyclometric complexity (CC):

    CC < 10: "simple" code
    10 > CC < 20: "moderately" complex code
    20 > CC < 50: "severely" complex code
    CC > 50: "untestable"

He based that experience on a simple equation of:

    # of test cases = 2CC-1

Which suggests code with a CC as low as 15 will have 16,384 test cases!  Ouch!  That has got to hurt considering functional testing only gets you about 20% of test coverage.

I think another valuable observation provided was software reliability is inversely related to CC, the higher the CC, the less reliable the software.  The most eye-popping statistic from the talk was the speaker’s observation that if you can refactor your code to have a CC in the 10-15 range, an organization would reduce their maintenance costs by 2 to 5 times!

Finally, these are the ten measures from the talk that I thought I would share with all of you and how one should optimize the measures.  The words in the parenthesis are what level the measures speak to.

1. Dollars per service -> higher is better (corporate)
2. Service vitality index (new service revenue/total revenue) -> ~10%-20% is good (corporate)
3. Number of new services/total services -> higher % is better(management)
4. Mean time to service deployment -> lower t is better (management)
5. Mean time to service charge -> lower t is better (management)
6. Service availability -> higher % is better (management)
7. Service reuse -> higher % is better (project)
8. Cost of not using/stopping the service -> lower $ is better (project)
9. Service complexity -> lower CC is better (service)
10. Service QA confidence -> higher % is better (service)