Step Ten: Track Project throughout Development
Refining Estimates throughout Project
Estimating software size, cost, and schedule should be an ongoing process. Preliminary estimates may be required to bid a job or to initiate the development process, or you may need to conduct a cost-benefit or return-on-investment (ROI) analysis to evaluate a project’s feasibility.
Preliminary estimates are the hardest to develop and are the least accurate because of the incomplete nature of the information available and the other factors discussed.
You can improve the accuracy of a preliminary estimate by using the sizing methodology identified in Step 4 or by using two different estimation techniques and having your analysts normalize the differences. There will still be a significant risk in using the preliminary estimate to structure a project or to evaluate risk in the early stages of a project life cycle.
Once a project has started, you will need to complete more detailed estimates to accurately plan the project and throughout the conduct of the project you will need to monitor the actual effort and duration of tasks and/or phases against planned values to ensure you have the project under control.
Summary
Software cost estimation is a difficult process but a necessary part of a successful software development. You can help ensure useful results by adopting a process that is standardized and repeatable. Several of the steps we have discussed, particularly those that do not result directly in the production of the estimate (Steps 1, 6, and 7) are often deferred or, worse still, not performed at all, often for what appear to be good reasons such as a lack of adequate time or resources or a reluctance to face the need to devise a plan if a problem is detected. Sometimes you simply have more work than you can handle and such steps don’t seem absolutely necessary. Sometimes management is reluctant to take these steps, not because the resources are not available, but because managers do not want to really know what they may learn as a result of scoping their estimates, quantifying and analyzing risks, or validating their estimates. This can be a costly attitude, because in reality every shortcut results in dramatic increases in project risks.
Previous:
Step Nine: Document Estimate and Lessons Learned
Step Nine: Document Estimate and Lessons Learned
Each time you complete an estimate and again at the end of the software development, you should document the pertinent information that constitutes the estimate and record the lessons you learned. By doing so, you will have evidence that your process was valid and that you generated the estimate in good faith, and you will have actual results with which to calibrate your estimation models. Be sure to document any missing or incomplete information and the risks, issues, and problems that the process addressed and any complications that arose. Also document all the key decisions made during the conduct of the estimate and their results and the effects of the actions you took. Finally, describe and document the dynamics that occurred during the process, such as the interactions of your estimation team, the interfaces with your clients, and trade-offs you had to make to address issues identified during the process. Cost models, which are based on the actual costs of past projects, can be calibrated and their accuracy can be demonstrated by comparing the costs of your current estimates with both past project data and the actual costs of your completed project, thereby adjusting the model input parameters to improve future accuracy.
Step Eight: Generate A Project Plan
The process of generating a project plan includes taking the estimate and allocating the cost and schedule to a function and task-oriented work breakdown structure. Models such as SEER Client for Microsoft perform this function automatically. The eight major software development phases
are: (1) concept, (2) acquisition, (3) requirements, (4) design, (5) code and unit test, (6) integration, (7) acceptance, and (8) post deployment.
Determining Costs from Effort Estimates
At this point in the estimation process, you should have a reasonably accurate projection of your project’s size and required effort, that is, an estimate of the number of person-hours by component and a sum of these projections, and you can now begin to price the estimate. As software estimation models generally account only for costs related directly to development, you may need to translate the required effort to a cost and finalize the estimate by adding in essential nonlabor costs. You can do so by answering the following questions:
What types of individuals do I need and when do I need them?
Identify the specific personnel requirements by task area (direct software management, software systems engineering, design, programming, configuration management, quality assurance, etc.). Develop a strawman schedule from the work breakdown structure or a staffing plan such as the one produced by SEER-SEM.
How experienced do they have to be? Assign specific staff levels to the task requirements and identify the level of experience required to satisfy the task. Some of the automated cost models will identify the tasks and develop a task-based schedule, which will minimize but not eliminate all of the work required to produce the software development plan.
Development Teams Using Note Cards: 20th Century Here Today
Lee Fischman, Galorath Guru attended the PMI conference last week. He sent an email during the confernce concerned about note cards as a project management approach. Lee sent the following for publication:
“These Agile teams that use note cards claim all sorts of wonderful
benefits from them, like seeing the entire project’s status on a wall,
and solid ownership of tasks. In fact, case tracking systems like JIRA
and Fogbugz do all that too, plus lots more such as sorting, metrics,
etc. I guess one reason people like note cards is that they are so
tangible. But using them reminds me of people who print out their
emails - soooo 19th century. Note cards are even worse, because they
are hand-written. Agile proponent Alistair Cockburn called these
“Information Radiators” but there doesn’t seem to be strong theoretical
support for the idea of using note cards over online systems. It’s not
that the entire Agile community feels like they have to do these things
by hand, because Rally, VersionOne, XPlanner and other vendors do offer popular tools to automate project management. Apparently other people agree with the notion that note cards are, uh, not optimal”
Bad Assumptions Invalidate Business Value and ROI Analysis
I recent email announced “ROI estimates in business fail primarily because managers give too much attention to the “payout” odds and too little attention to measuring and managing “probability” odds. A good risk and sensitivity analysis of the assumptions behind the predictions allows you to do both. ”
This is so true. Inappropriate or unlikely assumptions looking for payout based on the absurd (such as assuming that saving 1 minute per day of employees filling in their time card has a huge savings in a year) and a myriad of other traps cause business value and ROI to be appropriately produced using sound business case analysis.
In the same vein, absurd assumptions can cause cost and schedule estimates to be low. SEER evaluates the viability of assumptions, helping produce estimates that can actually be deployed.
Step Seven: Estimate Validation and Review
Step Seven: Estimate Validation and Review
At this point in the process, your estimate should already be reasonably good-but it may not be as good as you think. It is important to verify your methods and your results in a step called validation, which is simply a systematic confirmation of the integrity of an estimate. By validating the estimate, you can be more confident that your data is sound, your methods are effective, your results are accurate, and your focus is properly directed.
It may be tempting to skip this review due to a lack of time, personnel or budget, and there is the unappealing possibility that a close examination may reveal faults in the logic of your process. Nevertheless, the costs involved in performing a proper validation will be dramatically less than the cost overruns that are likely to develop during a poorly managed software project.
There are many ways to validate an estimate. Both the process used to build the estimate and the estimate itself must be evaluated. Ideally, the validation should be performed by someone who was not involved in generating the estimate itself, who can view it objectively. The analyst validating an estimate should employ different methods, tools and separately collected data than were used in the estimate under review.
Audits Improve Processes
I know not many people like an audit. Brings up thoughts of people trying to show you made a mistake.
But I just received a notice of our upcoming ISO 9001:2000 audit and I am thrilled. While we deal with processes each day, it is productive to take some time and ensure we are doing out best and to look for ways to do things later.
Part of proper management is reducing waste, seeing things that are done that don’t add value and eliminating them.
Another part of such an audit is seeing where people are doing well and making sure the whole company uses that best practice.
Initially the goal is just to have a process, even if it isnt the best. Then to refine the process(es) to reflect best practices.
At yesterday;s best practice seminar someone discussed waste, and looking to eliminate waste within the organization. I look at some of the simplest things many organizations do that are waste and can be eliminated… Pens handed out at conferences (do we think people don’t show up with writing implements, or that the cheap pens are going to win any gratitude.) Whether you are dealing with CMMI, ISO 9000, ITIL, or whatever process approach.. look for ways to eliminate waste. And remember “the enemy of the best is the good”
2 Kinds Of Metrics: Status & Trend and Effectiveness
First the obvious:
: Status / Trend Metrics: e.g. productivity, defects removal rate, cost, schedule
Most important for improvement: Effectiveness ( 5 max)
“What we are doing that we should not do” e.g. number of delivered critical defects
“What we are not doing that we should do” e.g. number of defects that got past inspections
- These metrics may change over time as we improve
- These are the most important for improvement
Step Six: Quantify Risks and Risk Analysis
The best managers of software projects seem to have an uncanny ability to anticipate what can happen to their projects and devise just-in-time mitigation approaches to avoid the full impacts of the problems. In reality, this ability is simply the skillful application of well known risk management techniques to the well known problems of software management. Unfortunately, too many software managers are skilled in seeing potential risks and then ignoring them outright.
Before we explore the risk management process and how to apply it to the risks associated with sizing and estimation, it is important to understand what a risk is and that a risk, in itself, does not necessarily pose a threat to a software project if it is recognized and addressed before it becomes a problem.
Oh Glorious Day.. Weapons System Decisions To Be Made Based On Total Life Cycle Costs
The October 13 2008 issue of Federal Times has a story Titled “Better Estimates Urged for Weapons Programs” about John Young, Pentagon Acquisition Executive and his movement for “establishing life cycle metrics early in the acquisition process”
Galorath has been preaching evaluating total ownership costs, not just development, for years. And we understand the pressures to “get it out in the field to support the warfighter” Using all the SEER products such as SEER for Hardware, Electronics and Systems support total ownership costs and tradeoff analysis to determine the most effective system design and architecture from a set of alternatives.
We all realize that there are other issues beyond just total ownership cost to keeping our country and those in harms’ way safe. And I know that combining estimating of total ownership costs early and often can support that goal.
