Wall Street Journal Article Points Out Monte Carlo Inadequacies
This article could have been written by Galorath’s Evin Stump who warns of Monte Carlo’s inadequacy in capturing highly improbable events.
Note…. I am not saying Monte Carlo is of no value… on the contrary… I believe Monte Carlo analysis is extremely useful. But is is not so good at capturing the risk of the highly improbable.
The WSJ article on risk and Monte Carlo Analysis Inadequacies pointed out:
“there is little chance your Monte Carlo simulation would have highlighted a scenario like the market slide just seen. Though these tools typically run a portfolio through hundreds or thousands of potential market scenarios, they often assign minuscule odds to extreme market events.
These models were supposed to help quantify and manage the risks of mortgage-backed securities, credit-default swaps and other complex instruments. But given the events of the past couple of years, it appears that the models often gave big institutions, as well as small investors, a false sense of security.
Also controversial is that many Monte Carlo simulations assume that market returns fall along a bell-curve-shaped distribution. That means a high probability may be assigned to, say, a stock-market return of 5%, which would fall toward the middle of the bell, and negligible odds assigned to a 54% decline, which would fall near the extreme edge, or “tail.”
“In a bell-shaped curve the probability of getting one of these extreme outcomes we’re seeing is basically zero,” said Paul Kaplan, vice president of quantitative research at Morningstar Inc.
While a bell-curve model indicates there is almost no chance of a greater than 13% monthly decline in the Standard & Poor’s 500-stock index, such declines have happened at least 10 times since 1926, according to a report by Mr. Kaplan.”
SEER’s new XIPRR project removes the issues of Monte Carlo analysis and take into account “Black Swans”… the highly improbable. I will be presenting Evin’s paper on XIPPR at the SCEA conference in Saint Louis in June.
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.
Related posts:
- Estimating Project Risk Using Scenarios (Without Monte Carlo) I was at a costing syposium today where Paul Garvey of Mitre gave a talk on doing risk analysis without...
- Black Swans: The Impact of the Highly Improbable Before the exploration of Australia by Europeans, people thought all swans were white. This assumption seemed completely confirmed by empirical...
- The Risk of Statistical Risk I was at the SCEA (Society of Cost Estimation and Analysis) conference this week. Some of the buzz was about...
- C language inventor dies I just heard this… Another great man in computing passed away.. While the popular and other press has been pre-occupied...
- GAO Knowledge Points A valuable GAO document discusses knowledge points.. the certainty that systems are appropriate. “Because military programs tend to start...
Comments
3 Responses to “Wall Street Journal Article Points Out Monte Carlo Inadequacies”
Leave a Reply
Project Prediction Infographic
Actually, as the article goes to point out, nothing is wrong with Monte Carlo. What’s wrong are the distributions used and the assumptions made.
I think there are two important aspects of this problem. The first problem is the fact that the Monte Carlo method itself has difficulties in simulating rare events. These difficulties can be partially overcome by replacing the bell curve with an asymmetric risk function. But the fundamental solution to this problem lies in the successful modeling isolated phenomena themselves.
The second problem relates to using the normal curve for the simulation of risk, and I think that this is due to incorrect interpretation of the central limit theorem. Using the bell curve implies that each one of the sequential random events has a finite variation. But this assumption is characteristic only for the simple human actions (see my paper “Human Effort Dynamics and Schedule Risk Analysis” here: http://www.pmforum.org/library/papers/2009/PDFs/mar/Human-Effort-Dynamics-and-Schedule-Risk-Analysis.pdf).
In this case of simple human actions we deal with the central limit theorem for finite variations and correspondingly with the bell curve as a risk function.
For difficult human actions we deal with another version of the central limit theorem for the sequential events with infinite variations. In this case the risk function is a fat tail distribution. But there are many such distributions, characteristics and behaviors are radically different from each other. This means that each specific problem must have its adequate function of risk in the form of the specific fat tail distribution. For instance, for the schedule risk analysis this fat tail distribution is the Cauchy distribution, which can be derived in an analytical way (see the same paper). This means that the problem reduces to the derivation or finding in some other way the analytical form of an adequate fat tail distribution for the specific task.
Pavel Barseghyan
There is no reason for all the fancy math. All Monte Carlo simulators come with built in PDFs. Triangle, Beta, or arbitrary “shapes” can be used for each event or activity if you are modeling a scheduling.
Simply adjust the right side of the PDF to represent. The Cauchy distribution would not be a good choice, because it too us symmetric.
One failure in the financial domain was (and possibly still is) the “return to the mean of a symmetric distribution. Changing from Gaussian to Cauchy does not solve this problem.
Please read the Big Short and All The Devils in One Place and the source papers from the University of Chicago to the proper background on this complex issue.
There is no reason to “derive” all this analytical models. These are non stationary stochastic process with no closed form solution. There is no assurance that the general problem of models even has an analytical form – this is the fundamental motivation for MCS since these class of problems were initiated in the late 40′s.
The “some other way” is a MCS model using several types of activity networks – Markov transition, PETRI nets or combination. The Crystal Ball tool is a common modeling tool for this area.