Archive for the ‘Measures’ Category

• Musings on (Agile) Metrics

  Date: 2010.10.18 | Category: Agile, Conferences, Design Excellence, Design for Six Sigma, Lean, Measures, Scorecards | Response: 0

  Last week I attended a really interesting session at Agile Open NoCal about what are good metrics for Agile teams.  Since I spent a lot of time thinking about this when working with Cardinal Health on their Design Excellence program, here are some thoughts I have.

  Metrics are a powerful tool to influence behavior.  They should be applied judiciously and cautiously to any problem since they ALWAYS create unintended and unexpected behavior.  The larger the organization, the more important it is for measurement since that is the way big organizations receive feedback and can adjust their behavior.  However, it is unreasonable to insist all metrics are destructive and irresponsible to expect a large business to fly blind without some sort of data.

  I tend to think, most people do not understand how to design good metrics or they create metrics that are easy to gather, i.e. using time sheets or something else similarly foolish, or they pick metrics which are valuable to the data collector, not anyone else.  I don’t claim to be an expert, but I have some ideas which I think are consistent with Lean Thinking and Agile.  There are four guiding principles I can provide:

  1. The only metrics that matter are the one tied to customer value or satisfaction.  All the rest are intermediate measurements.
  2. The people being measured should have a say in creating the measurement system on which they are judged.
  3. Apply the Lean concept of Measure Up when creating your metrics.
  4. The best metrics are temporary.  Once the metric has produced the desired outcome, discard it.

  Pascal Dennis in his book, Getting the Right Things Done (not my favorite book – long on story, short on specifics), gives some great guidance on the first principle when flushing out the concept of True North. True North is an emotional description of the strategic and philosophical goals of the organization tied to real-life, customer-focused metrics.  Not the kinds of metrics middle managers care about or software teams, but the types of metrics that show up in reports to shareholders and are on the minds of the executives and senior leaders.  These are the metrics EVERYONE in the organization needs to be focused on and thinking about every day.

  In my experience, the people doing the work know what metrics are meaningful to them and which have no impact on their day-to-day activities.  To create a powerful measurement system that gives accurate, real-time data, engage the people doing the work in creating this system.  In addition, the metrics they come up with, are the metrics on which they are evaluated.  The common objections one runs across when developing this idea with middle manager are “what if the workers don’t come up with anything?” or “what if they are not sophisticated enough?”  I feel if you explain to the workers what is the True North, the main business metrics, they will come up with meaningful metrics that tells them if their work is contributing to the objectives of the organization.  The people who work in the company want the business to succeed (and if you feel otherwise, you have a different, more serious problem).

• Agile Control Charts

  Date: 2008.12.08 | Category: Agile, Design Excellence, Design for Six Sigma, Measures | Response: 0

  I stumbled across Agile Shrugged today and read a post on how the author experienced a lot of variation in their team velocity.   The author did not seem too concerned with all the “sawtooth” behavior of velocity chart because they felt the velocity graph was a “fairly accurate measure of our teams velocity”.  Let’s see if this qualitative assessment matches up with statistical process control, one of the Six Sigma tools.

  In Six Sigma, a common tool Green Belts and Black Belts use to monitor their processes are control charts.  The control chart is an objective measurement tool to determine if the process under inspection can reliably produce what is expected of it in the future.  By examining the patterns of the data on the control chart we can understand if the process is “in control” or “out of control”.  If the process is “out of control”, then we need to inspect the process, discover the cause of the variation, “mod” the process and monitor the changes with…a control chart.

  The diagram below is the velocity data from the team.  You can see the “sawtooth”, or oscillation, in velocity over the course of ten iterations.  The team does seem to have improved their velocity over time, but we still have an order of magnitude difference in the completed output of the team from week-to-week.  Is this significant?  Should we care about the oscillation?

  

  To make a control chart, we need to run some basic statistics on the data: compute the mean, standard deviation (σ) and determine the upper and lower spec limits (mean ± 3σ).  Here are the results for this data set:

   mean: 6.6 ideal days    upper spec limit (USL): 24 ideal days
   σ:    5.8 ideal days    lower spec limit (LSL): -10.8 ideal days

  Next, we will plot the data on a control chart showing the mean (green line), the USL (red line), 1σ and 2σ (purple lines).  I have eliminated the LSL because a negative velocity is meaningless.

  

  Now here is the interesting part: looking at the patterns.  Using the Nelson Rules, we can examine the patterns in the control chart to see if the process is “in control” or “out of control”.  Here are the rules that I think are relevant for this chart (please review all the rules if you are interested in learning more):

  1. One point is more than 3 standard deviations from the mean. Nope
  2. Fourteen (or more) points in a row alternate in direction, increasing then decreasing. Only 8 points so far; maybe something to watch out for.
  3. Four (or five) out of five points in a row are more than 1 standard deviation from the mean in the same direction.  Only 2 out of 5 data points.

  What do we find by creating and examining the control chart?  Statistical confirmation that our qualitative opinion that our planning process is indeed fine.  Great!  Keep up the good work!

• Function Points as an Agile Metric

  Date: 2008.12.02 | Category: Agile, Measures | Response: 0

  Dave Nicolette posted an interesting blog post about using function points (FP) as a means to compare productivity between IT projects using different processes.  I understand the concept – find a metric that speaks to the audience we are trying to communicate with that allows an apples-to-apples comparison.  However, why do I find this still an unsatisfactory answer to the productivity issue?

  I tend to think this metric would only be useful for academic types or a research paper.  Sure, it is create more evidence that “Agile is great and waterfall is bad”, and how long do we have to wait for this evidence?  One business cycle?  Since I am on a roll smashing your good idea, I also think the barrier for entry is too high for people wishing to make the case Agile would be good for their organization.  Not only do they need to understand Agile practices & techniques, they now need to become FP counters.  Now we need to start counting FP on all our IT projects?  Is this something our industry has the skill to do?  The evidence I have seen shows we do not.

  Yes, FP are “good” because ISO has made them the standard, there is clear way to count them (for the people who know) and academics (and attornies) love them.   There are real reasons why FP are not used in US business today.  I work with the ideal organization (large Fortune 500 company) for FP analysis, they are always hungry for metrics and they do not use FP.  I wonder why that is the case?  It is not due to a lack of education.  There are well-placed, knowledgable individuals aware of this metric, so there must be something else preventing FP.  We need to understand why people who know about this metric, can afford the consultants to do the analysis and still do not use the metric.  When we overcome those issues, then let’s talk about pushing this metric on Agile teams.  In the meantime, let’s stick with “basics” like throughput, takt time or use EVM (as much as I hate it – it is comprehensible to business folks).

• SOA Measures for Success

  Date: 2008.10.28 | Category: Conferences, Measures | Response: 0

  One of the sessions I visited today was about what are the correct measurements to determine if your SOA initiative was successful.  One of the most interesting statistic from the speaker was this continuum of cyclometric complexity (CC):

      CC < 10: "simple" code
      10 > CC < 20: "moderately" complex code
      20 > CC < 50: "severely" complex code
      CC > 50: "untestable"

  He based that experience on a simple equation of:

      # of test cases = 2CC-1

  Which suggests code with a CC as low as 15 will have 16,384 test cases!  Ouch!  That has got to hurt considering functional testing only gets you about 20% of test coverage.

  I think another valuable observation provided was software reliability is inversely related to CC, the higher the CC, the less reliable the software.  The most eye-popping statistic from the talk was the speaker’s observation that if you can refactor your code to have a CC in the 10-15 range, an organization would reduce their maintenance costs by 2 to 5 times!

  Finally, these are the ten measures from the talk that I thought I would share with all of you and how one should optimize the measures.  The words in the parenthesis are what level the measures speak to.

  1. Dollars per service -> higher is better (corporate)
  2. Service vitality index (new service revenue/total revenue) -> ~10%-20% is good (corporate)
  3. Number of new services/total services -> higher % is better(management)
  4. Mean time to service deployment -> lower t is better (management)
  5. Mean time to service charge -> lower t is better (management)
  6. Service availability -> higher % is better (management)
  7. Service reuse -> higher % is better (project)
  8. Cost of not using/stopping the service -> lower $ is better (project)
  9. Service complexity -> lower CC is better (service)
  10. Service QA confidence -> higher % is better (service)