What You Don’t Measure Can Hurt You
Following up on a new year’s resolution I brought a scanner into my office. I was determined to scan what was necessary and throw away what was not. Based on the findings at the bottom of my in-basket, I generally went electronic sometime in 2003.
As I was throwing away old copies of IEEE Software I noticed an article entitled “What You Don’t Measure Can Hurt You.” With such a title and since I have had it in my in box since 2003 I just had to take a break and read it. Its points are as valid today as they were in 2003:
“In 1924, Walter Shewhart adapted statistical methods to the problem of quality control in the manufacturing sector.� In manufacturing, the observed and actual number of defects is not significantly different. In software development these two numbers routinely vary significantly. Contributing causes for extreme variation in software measurement include the following:
- People are the software development process
- Software measurement might introduce more variation than the process
- Size metrics do not count discrete, identical units
- SPC must be adapted to software to provide a tool that isolates variation requiring corrective action from variation that is unavoidable and random
One of the paper’s great examples is: if we assign a smaller team the inspection may find fewer defects.� That doesn’t equate to higher quality software. So measure both and be careful not to draw erroneous conclusions.
Thankfully IEEE Software is available electronically. Here is the article “What You Don’t Measure Can Hurt You”
PS: I found a candy bar in the bottom of that in-basket too. Scary.
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.
Related posts:
- You Can Manage What You Can’t Measure It was good to see Tom DeMarco’s article in IEEE Computer magazine renouncing his famous statement, “You can’t manage what...
Comments
One Response to “What You Don’t Measure Can Hurt You”
Leave a Reply
Project Prediction Infographic
I’m not sure we would want to measure the people directly in any sense. I think that would be the equivalent to measuring manufacturing process when saying “people are the software development process.” That would make little sense and I doubt anyone ever considered that approach.
In a real world example, we had a small QA team assigned to “double check” our defect prioritization process. They were to see if we were “accurately” identifying the priority assigned to each defect. No methodology was used, just the joint opinion of the assigned team (I guess is was a wisdom of crowds/delphi technique).
They would send me a list of defects each day that they considered mis-prioritized. Every day, we got a list of defects we needed to adjust the priorities on, and we did.
I did an analysis to compare their “new priorities” and the final disposition of those defects to those defects that were not adjusted.
In a nutshell, what they were doing was indistinguishable from randomly selecting defects and just assigning a higher priority (as opposed to finding truly under-prioritized defects).
One of the members of the group admitted that they felt they must always find something in the hours they spent each day in this review, otherwise they would not be seen as doing their job. I jokingly suggested we replace the team with a random selection process, but nobody seemed amused (saving ~10 people salary x ~2 hours a day …).
We could tell the added QA process was ineffective because we were measuring the defect repair process and understood how it performed. Many folks don’t get it that this is possible in many cases and works very well (and we were not measuring people per se, but the overall repair process).
Thanks for provoking the memory.
Bruce Benson
http://PMToolsThatWork.com