Estimating Project Risk Using Scenarios (Without Monte Carlo)
I was at a costing syposium today where Paul Garvey of Mitre gave a talk on doing risk analysis without using Monte Carlo or other statistical techniques. Paul, one of the gurus of statistical risk and the author on the definitive book on cost risk: “Probability Methods for Cost Uncertainty Analysis”, said after 25 years of dealing with Monte Carlo and all the mistakes the non-gurus make in setting up Monte Carlo analysis he had determined it is prudent to use a non statistical technique.
Paul discussed how well meaning but non-gurus would setup Monte Carlos and provide correlations in programs like Crystal Ball or @Risk that had already been accounted for, hence double counting correlation. And woudl apply distributions without really understanding the range. And how where they were all done he didn’t know what them impact might be.
Paul then told of his work in estimating risk by identifying a scenario that might actually occur, the costing the problem with that scenario’s set of circumstances as the risk estimate on top of the likely estimate that had been produced.
Paul is loooking towards others in the systems engineering community to actually identify the methodology for defining the scenarios.. .not the worst case, not the best case, but the series of events that might occur that would cause a $1 estimate to end up as a higher number ($1.75 for example if the scenario occured) Pauls also discussed scenario protection.. that is the cost of ensuring the scenario did not occur.
THe beauty of this approach is that the complicated and error prone statistical approaches are avoided and the resulting risk number can be explained to a program manager or others easier than trying to explain an S curve. And there would likely be fewer errors.
I must admit I was scratching my head about how to develop a scenario. We have used risk registers with their series of events that occur within a risk item but this didn’t seem to be a scenario. I asked the question but it was deferred until others did some work on it. Afterwards a member of the audience told me the scenario problem had been solved. Peter Schwartz’sbook The Art of the Long View: Planning for the Future in an Uncertain World. describes the process. According to Wikipedia “The art of the Long View (Doubleday, 1991) is considered by many to be the seminal publication on scenario planning, and is used as a textbook by many business schools. He also co-wrote The Long Boom (Perseus, 1999), a look at a future characterized by global openness, prosperity, and discovery. that discusses how to develop scenarios.”
The second half of Paul’s talk showed how to apply some statistics to this approach to provide a feeling or probability of an answer.
I spoke with Steve Book at lunch and Steve remained convinced that statistical risk is the best. Of course Paul likes statistical risk best too, when the analyst understands it. But there is something I like about Paul’s approach. Understandable, an artifact of the systems engineering process.
When Paul’s presentation is posted I will provide a link here.
More on Scenarios
According to Schwartz “Scenarios are narratives of alternative environments in which today’s decisions may be played out. They are not predictions. Nor are they strategies. Instead they are more like hypotheses of different futures specifically designed to highlight the risks and opportunities involved in specific strategic issues. To be an effective planning tool, scenarios should be written in the form of absorbing, convincing stories that describe a broad range of alternative futures relevant to an organization’s success.” Steps include:
- Brainstorming Key Factors
- Distinguishing Pre-Determined Elements from Uncertainties
- Establish “The Official Future” What stakeholders believe will actually happen
- Building Narratives.. How the world will get from its as is state to the predicted future,
Then identifying Analysing Such As:
- Winners vs losers
- Crisis Vs Response
- Good News / Bad News
- Wild Cards
He points out that decision makers must own the scenarios
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page, call us at +1 310 414-3222 or click a button below to ask sales questions, sign up for our free library or schedule a demo.