In a real world example, we had a small QA team assigned to “double check” our defect prioritization process. They were to see if we were “accurately” identifying the priority assigned to each defect. No methodology was used, just the joint opinion of the assigned team (I guess is was a wisdom of crowds/delphi technique).

They would send me a list of defects each day that they considered mis-prioritized. Every day, we got a list of defects we needed to adjust the priorities on, and we did.

I did an analysis to compare their “new priorities” and the final disposition of those defects to those defects that were not adjusted.

In a nutshell, what they were doing was indistinguishable from randomly selecting defects and just assigning a higher priority (as opposed to finding truly under-prioritized defects).

One of the members of the group admitted that they felt they must always find something in the hours they spent each day in this review, otherwise they would not be seen as doing their job. I jokingly suggested we replace the team with a random selection process, but nobody seemed amused (saving ~10 people salary x ~2 hours a day …).

We could tell the added QA process was ineffective because we were measuring the defect repair process and understood how it performed. Many folks don’t get it that this is possible in many cases and works very well (and we were not measuring people per se, but the overall repair process).

Thanks for provoking the memory.

Bruce Benson
http://PMToolsThatWork.com