A lot of the calls I take focus on metrics for the QA team. How many testers per developer, how many hours of testing, how many defects, etc. Unfortunately many of these metrics don’t end up yielding useful information often because they are too simple and they leave to many variables open. We like metrics because they can help us get a read on project directions and if we are falling in the averages or not but it is too easy to fall into the trap of easy metrics that mislead. I discussed this in earlier research on defect containment (see Toolkit: Defect Containment and Quantitative Defect Management, G00160257). Yesterday at IBM’s Innovate conference Ivar Jacobson talked about the need to focus on metrics that were better not easy:
People look at the wrong things:
– Things that are easy to measure but meaningless
– “Checking boxes” rather than results
– Few, if any, qualitative measures
And present them in the wrong way:
– Confusing and contradictory data
– Meaningful only to specialists – and maybe not even to them!
– Unclear purpose
– Point data and not trends
– Hard to understand and use to make decisions
To close the gap, how well you are doing needs to be measured in business terms
He suggested looking at metrics that help see if we are delivering better applications, delivering faster, systems that are easier to use, cheaper to run, and leading to happier users. These are not single dimension views but they focus on what matters. IBM is adding a number of capabilities for improved metrics to its Team Concert ALM product (as are other ALM vendors) and while these tools do provide increased reporting capabilities it will be important to not misuse the information and to recognize that seeing bad numbers doesn’t mean seeing "bad people". This is the real core of the use of metrics is while they may be the proverbial warning flag they don’t tell you directly how to fix the problem. This doesn’t mean all dashboards are evil but they can’t replace Management by Walking Around and getting your hands dirty.
The Gartner Blog Network provides an opportunity for Gartner analysts to test ideas and move research forward. Because the content posted by Gartner analysts on this site does not undergo our standard editorial review, all comments or opinions expressed hereunder are those of the individual contributors and do not represent the views of Gartner, Inc. or its management.