Make or Break: Why Accurate Cost Estimation Is Key
A recent article from executivebrief.com sent to me By Dr. Ricardo Valerdi of MIT discusses how the accuracy of the cost estimation process can make or break a project’s success.
“When it comes to controlling costs, it is a critical first step to make appropriate estimations at the outset of a project. Being able to control costs is largely a matter of adhering to established guidelines, oftentimes by learning from previous projects and reacting to current circumstances efficiently and effectively.”
It is amazing to me how simple this concept is, yet how many CEO’s don’t even know it is possible. Accurate estimates (of course with risk and uncertainty factored in) yield viable project plans which can then be successfully monitored and controlled. In fact the application of techniques such as earned value management, as powerful as they are, crumble when the baseline plan is unachievable.
Step Ten: Track Project throughout Development
Refining Estimates throughout Project
Estimating software size, cost, and schedule should be an ongoing process. Preliminary estimates may be required to bid a job or to initiate the development process, or you may need to conduct a cost-benefit or return-on-investment (ROI) analysis to evaluate a project’s feasibility.
Preliminary estimates are the hardest to develop and are the least accurate because of the incomplete nature of the information available and the other factors discussed.
You can improve the accuracy of a preliminary estimate by using the sizing methodology identified in Step 4 or by using two different estimation techniques and having your analysts normalize the differences. There will still be a significant risk in using the preliminary estimate to structure a project or to evaluate risk in the early stages of a project life cycle.
Once a project has started, you will need to complete more detailed estimates to accurately plan the project and throughout the conduct of the project you will need to monitor the actual effort and duration of tasks and/or phases against planned values to ensure you have the project under control.
Summary
Software cost estimation is a difficult process but a necessary part of a successful software development. You can help ensure useful results by adopting a process that is standardized and repeatable. Several of the steps we have discussed, particularly those that do not result directly in the production of the estimate (Steps 1, 6, and 7) are often deferred or, worse still, not performed at all, often for what appear to be good reasons such as a lack of adequate time or resources or a reluctance to face the need to devise a plan if a problem is detected. Sometimes you simply have more work than you can handle and such steps don’t seem absolutely necessary. Sometimes management is reluctant to take these steps, not because the resources are not available, but because managers do not want to really know what they may learn as a result of scoping their estimates, quantifying and analyzing risks, or validating their estimates. This can be a costly attitude, because in reality every shortcut results in dramatic increases in project risks.
Previous:
Step Nine: Document Estimate and Lessons Learned
Step Nine: Document Estimate and Lessons Learned
Each time you complete an estimate and again at the end of the software development, you should document the pertinent information that constitutes the estimate and record the lessons you learned. By doing so, you will have evidence that your process was valid and that you generated the estimate in good faith, and you will have actual results with which to calibrate your estimation models. Be sure to document any missing or incomplete information and the risks, issues, and problems that the process addressed and any complications that arose. Also document all the key decisions made during the conduct of the estimate and their results and the effects of the actions you took. Finally, describe and document the dynamics that occurred during the process, such as the interactions of your estimation team, the interfaces with your clients, and trade-offs you had to make to address issues identified during the process. Cost models, which are based on the actual costs of past projects, can be calibrated and their accuracy can be demonstrated by comparing the costs of your current estimates with both past project data and the actual costs of your completed project, thereby adjusting the model input parameters to improve future accuracy.
Step Seven: Estimate Validation and Review
Step Seven: Estimate Validation and Review
At this point in the process, your estimate should already be reasonably good-but it may not be as good as you think. It is important to verify your methods and your results in a step called validation, which is simply a systematic confirmation of the integrity of an estimate. By validating the estimate, you can be more confident that your data is sound, your methods are effective, your results are accurate, and your focus is properly directed.
It may be tempting to skip this review due to a lack of time, personnel or budget, and there is the unappealing possibility that a close examination may reveal faults in the logic of your process. Nevertheless, the costs involved in performing a proper validation will be dramatically less than the cost overruns that are likely to develop during a poorly managed software project.
There are many ways to validate an estimate. Both the process used to build the estimate and the estimate itself must be evaluated. Ideally, the validation should be performed by someone who was not involved in generating the estimate itself, who can view it objectively. The analyst validating an estimate should employ different methods, tools and separately collected data than were used in the estimate under review.
Step Four: Software Sizing
Size is generally the most significant (but certainly not the only) cost and schedule driver. Overall scope of a software project is defined by identifying not only the amount of new software that must be developed, but also must include the amount of preexisting, COTS, and other software that will be integrated into the new system. In addition to estimating product size, you will need to estimate any rework that will be required to develop the product, which will generally be expressed as source lines of code (SLOC) or function points, although there are other possible units of measure. To help establish the overall uncertainty, the size estimate should be expressed as a least-likely-most range.
Predicting Size
Whenever possible, start the process of size estimation using formal descriptions of the requirements such as the customer’s request for proposal or a software requirements specification. You should reestimate the project as soon as more scope information is determined. The most widely used methods of estimating product size are:
GAO Knowledge Points
A valuable GAO document discusses knowledge points.. the certainty that systems are appropriate. “Because military programs tend to start product development with more unknowns, it takes them additional time, sometimes until well after production begins, to actually discover and capture enough solid information to attain full product knowledge and thereby virtually eliminate risk.” Read more
Estimating Project Risk Using Scenarios (Without Monte Carlo)
I was at a costing syposium today where Paul Garvey of Mitre gave a talk on doing risk analysis without using Monte Carlo or other statistical techniques. Paul, one of the gurus of statistical risk and the author on the definitive book on cost risk: “Probability Methods for Cost Uncertainty Analysis”, said after 25 years of dealing with Monte Carlo and all the mistakes the non-gurus make in setting up Monte Carlo analysis he had determined it is prudent to use a non statistical technique. Read more
“But the Product Meets The Specs…” Poor Analysis, Cost Overruns and Risk
I recently got the clever idea to purchase a folding bicycle so I could have a decent bike when on vacation… The plan was to check the bicycle with the airlines in a standard suitcase. I spent hours researching on the web. I spoke to a couple of bike stores. Then I placed my order. A few days later a BIG box appeared on my doorstep. The bicycle was a magnificent piece of engineering. Unfolds in about 15 seconds. Only one problem. The bicycle was designed for the older airline luggage size. With the newer, more stringent standards this bicycle is high risk. If they allow it to be checked at all there is up to a $270 extra charge.
The people at the bike store didn’t tell me. Perhaps they didn’t know. i wasn’t smart enough to catch the problem in my analysis activity.
It struck me that this is the same kind of problem that plagues product development and associated cost analysis. We have an end in mind. But that end gets lost in the details of specifications. And bad assumptions…. Parametric modeling can handle the cost, schedule, risk, etc. for product development. And it should identify risks and tie them back to the estimates. A model that can predict the overruns / inefficiencies in an estimate could potentially mitigate such problems. This is more than just Monte Carlo analysis which can show the risk within WBS elements, but cause and effect analysis. SEER approaches this by tying a risk register back to the estimate. And other, more exotic methods are floating around the labs. Our people have been showing this off in conjunction with Oracle to rave reviews.
Anyone want to buy a folding bike?
The Risk of Statistical Risk
I was at the SCEA (Society of Cost Estimation and Analysis) conference this week. Some of the buzz was about risk, both talks given at the conference and the ongoing risk arguments. For several years the risk gurus have been lining up to show how to do more robust risk analysis. While I would not say they are getting carried away I would say i get concerned with the differences of opinion and the numerous options provided by smart people.
One of my heroes in risk, Dr. Steve Book or MCR points out that risk analysis should include correlation.
One of my other risk heroes, Evin Stump (of Galorath), points out that defining correlation properly for a work breakdown of any size can involve of thousands of correlation entries. For example a 500 element WBS has 124, 500 correlations and a 1000 element WBS has 499,000 correlations. Dr. Book doesn’t point that out but he does say “use .2″ That solves the hundreds of thousands of correlations issue. But according to Evin that doesn’t provide more accurate risk analysis. Evin points out “if two or more risky items are not statistically independent, a Monte Carlo simulation that fails to account for their correlation will underestimate their combined risk” He then asks “what if you overestimate correlation?” Hmmm could it be that .2 correlations could overestimate some systems. Evin also points out how difficult it is to actually determine correlation…. For example, what is the correlation between a light bulb and a light bulb on/off switch… Probably near zero but most people would be tempted to assign a high correlation. Read more
Risk Analysis & Conference In the Netherlands
I spent this week with our international group… at two conferences (ISPA / SCEA and SSCAG) in the Netherlands.
There were several excellent papers given by Galorath people as well as many SEER users.
Risk analysis has always been one of my favorite features of SEER. And as the risk analyis community matured the risk gurus asked for the ability to fine tune the risk methodology. I was pleased to hear the results of a risk survey and to see how many people use SEER’s internal risk features. SEER includes internal risk analysis and, for those who want to fine tune risk even more, SEER has an interface to Oracle’s Crystal Ball. The Crystal Ball interface allows users to specify more specific information regarding correlations and other risk details. The Oracle people were also at this conference and were very pleased with the SEER use of Crystal Ball. How are you using SEER’s risk features? Do you use SEER’s Crystal Ball interface?
