If your organization is like most, tough problems are addressed by assembling a group of SMEs into a meeting and hashing out a solution. Risk assessments are often performed in the same way: bring “experts” into a room, brain storm on the threats and hash out an agreed-upon set of vulnerability and impacts for each. I will leave the fundamental problems with scoring risks based on vulnerability and impact ratings for another post[1].
“None of us is as smart as all of us” is a common mantra. Certainly, we should arrive at better conclusions through the collective work of a number of smart people. We aren’t. Many people have heard the phrase “the wisdom of crowds” and implicitly understood that this reinforces the value of the collaborative effort of SMEs. It doesn’t.
The “wisdom of crowds” concept describes the phenomenon where a group of people are each biased in random directions when estimating some quantity. When we average out the estimates of the “crowd”, the resulting average is often very close to the actual quantity. This works with the estimates are given independently of one another. If the “crowd” collaborates or compares ideas when estimating the quantity, this effect isn’t present. People are heavily influenced by each other and the previously present array biases are tamped down, resulting in a estimates that reflect the group consensus and not the actual quantity being analyzed.
The oft cited example is the county fair contest where the crowd writes down his or her guess for the weight of a cow or giant pumpkin on a piece of paper, drops the paper in a box and hopes to have the closest guess to win the Starbucks gift card. Some enterprising people have taken the box of guesses and averaged them out and determined that the average of all guesses is usually very close to the actual weight. If, instead, the fair goers were somehow incentivized to work together so that they only had one guess, and if that guess were within, say 2 pounds of the actual weight, the entire crowd won a prize, it’s nearly a sure thing the crowd would lose every time, absent some form of cheating.
With this in mind, we should consider the wisdom of our risk assessment strategies.
[1] In the mean time, read Douglas Hubbard’s book: “The Failure of Risk Management”.