Improve Software Cost Estimating

General Dynamics
Ralph Porter and Joseph Lees

General Dynamics Electronic Systems, like other companies engaged in large-scale software and systems development, has grappled with the daunting "voodoo economics" of accurately estimating and controlling costs in the software development business. Yet, the company has evolved since the cost-overrun days of the 1970s and has undertaken many initiatives to improve financial projections that facilitate the delivery of high-quality systems to its customers, on time and on budget.

Major improvements came in the 1980s as Electronic Systems, previously part of GTE Government Systems, developed a formal process for collecting and harnessing historical financial and program data to more accurately project and control costs. Over the past two decades, the company has added a weapon to its cost-forecasting arsenal. It's called software metrics. The use of software metrics has improved Electronic Systems' ability to produce accurate software cost estimates and control costs throughout projects' life cycles.

Accurate software development estimates are critical for the company and its customers, considering Electronic Systems has been providing total systems solutions to the intelligence and defense electronics markets for over 40 years. The company has two major business organizations involved with leading-edge software development. The Intelligence Systems Organization is responsible for imagery exploitation and processing, data management systems, weather systems, war-gaming and intelligence analysis support, threat simulators, and virtual reality training systems. The Space and Information Operations Organization handles information operations, electronic warfare, intelligence analysis, and funded research in threat signals and networks.

Challenges of Software Estimation

Although critical to mission success, the job of estimating software costs is complicated by major challenges. First, the software and the processes and tools used to develop it are continually evolving. The development process is complicated by frequent changes in methodologies, languages, and standards. And, the system requirements, staff and resources are everchanging.

Prior to the 1970s, the process of estimating development costs was considered similar to a black art, and cost overruns on the order of 100 percent were common. Electronic Systems had its first experience with software metrics in the early 1970s with the creation of a set of "Blue Books" which included several volumes that specifically addressed standards for software and systems development processes, project planning, tracking, and metrics reporting.

This approach provided substantial improvements, but the broad mix of different projects handled by the organization limited the usefulness of such a simple approach. Electronic Systems' projects range from the very large, over one million lines of code, to the very small, under 2,000 lines of code. The company also handles formal projects, such as those with full DoD-STD-2167A, as well as informal projects based on best commercial practices. The relatively immature metrics used at this time were unable to capture these differences. By the late 1970s, Electronic Systems began utilizing its collection of historical data to substantially improve the estimation process and reduce cost and size overruns.

In 1981, Barry Boehm's landmark book, Software Engineering Economics, had a major impact on the company's processes. This book was one of the first to quantify the many factors influencing productivity of a software program. Some of the factors Boehm highlighted included the experience level of the analyst writing the initial requirements, the skill of the programmer performing the implementation, the programming language used in the application, and whether or not it was necessary to re-host to a target environment.

Electronic Systems' engineers began using spreadsheets to develop models that, almost from the beginning, significantly improved the accuracy of the estimation process. By adjusting model parameters, one can bind the estimates, and thus the risks, associated with a software program. It soon became apparent that more historical data would be critical to improving the accuracy of these models, and the organization instituted a formal process for collecting this information.

Role Throughout the Project Lifecycle

Since the early 1980s, the use of software metrics has taken on an important role throughout Through repetition, the model provides near-perfect results. project life cycles. Today, Electronic Systems employs metrics from the proposal phase to the post-development phase of  ach major project.

During the proposal phase, historical data feeds the overall software estimating process. When new metric data are introduced, the estimate is continually updated. Also, during the proposal phase, comparison data are used to validate estimates. For example, a historical  comparison that tracks productivity as a function of project size can be used to establish boundaries for estimating effort, schedule and staffing. The result of using these metrics? Cost estimates are no longer left to engineering judgment alone, and the company produces more realistic and reliable estimates. Except in rare circumstances, the organization's estimate of productivity rates on any project is likely to be within 20 percent of the organization's actual average productivity rate, when normalized for size.

During the program execution phase, planning is based on historical metric data and baseline estimates. The organization has collected the phase distribution of effort data and schedule for many of its programs and uses this trend data for estimating future programs.

Tracking and oversight are also based on metrics, and they provide early signs of problems and help evaluate risks in costs and schedules. Electronic Systems also uses a monthly software program review metrics chart to ensure consistency in data collection. The program review process allows for timely management oversight of projects' overall progress.

During the post-development phase, recently completed programs are analyzed. Actual data for size, effort, schedule, staffing, defects and defects fixed rates are collected. The final project data are then remodeled and compared to the original project estimate to assess what went right or wrong with the project and to capture "lessons learned." Recommendations are fed into the overall estimating process to improve future estimates, and the historical data are added to the corporate metrics database.

Spreadsheets vs. Commercial Tools

Spreadsheet-based models have improved Electronic Systems' cost-estimation accuracy in the 1980s. However, these models have their limitations. One problem with spreadsheet models is their inherent vulnerability to errors (e.g., data-entry errors) and this vulnerability increases along with the complexity of the model. Simply entering a number in the wrong cell could throw off the cost estimate for a large program by millions of dollars.

The spreadsheet approach also doesn't account for uncertainties. For example, what would the financial impact on a program be if a model that had a higher level of formality is followed? Would it increase costs by five percent or 30 percent? Spreadsheet models cannot manage these what-if scenarios.

In the late 1980s, Electronic Systems' engineers evaluated and began using commercial cost estimation tools. Eventually, the use of cost-estimating software largely replaced spreadsheet models. Electronic Systems frequently used SEER-SEM^TM, a software program from Galorath Incorporated (www.galorath.com), El Segundo, California. SEER-SEM accepts a wide range of input parameters, including those advanced by Boehm.

Parameters include size, personnel, complexity, environment and constraints, platform and application, development and acquisition  method, and standards. This tool also allows the creation of custom parameters. SEER-SEM's multiple knowledge bases address client-server, embedded, distributed, stand-alone systems, varied MIS applications, graphics, and signal processing environments.

In using SEER-SEM, the user can choose the probability level of estimates, and set different levels for each portion of the project. The software also provides detailed sensitivity analysis features that make it possible to determine the impact of adjusting specific project factors. This feature, for instance, makes it possible to quickly estimate the impact on cost if a project schedule is compressed by two months. A customer who is considering moving up the delivery date of the project could quickly change his or her mind after discovering that the change will cost $2 million.

The Importance of Calibration

Electronic Systems' engineers have discovered calibrating the results of a commercial cost estimation tool to actual program data can increase the estimate's accuracy. Two important parameters for the calibration process are the levels of formality for the specifications and the testing process. In general, the more formal the specifications and testing are, the lower the productivity.

For example, a very formal program run to military standards might require high-level designs, detailed designs, and test plans presented at stand-up presentations. These meetings might require a week of preparation and involve action items from the presentation needing to be addressed before taking the next step. A less formal approach, requiring only four hours of preparation, might involve technical interchange meetings where the project status is explained.

Calibration involves specifying the level of formality of a particular project so it matches the universe of projects contained in the cost estimation tool's database. By evaluating previous programs, estimators can provide optimum settings for these parameters. Typically program managers fill out forms addressing each parameter contained in a model. Estimators work with program managers to compare these parameters to those parameters that have been calibrated for previous projects.

Through repetition, the model provides near-perfect results. This process makes it possible to achieve highly reliable results on nearly every recent project, a level of accuracy considered an impossible dream only a decade ago. And, as we've seen in the financial evolution of the software business, the result is a blessing for Electronic Systems and its customers.

About the Authors

Ralph Porter is the Director of Process and Quality Assurance for General Dynamics Electronic Systems and Joseph Lees is a Senior Manager of Engineering responsible for all software estimating. Electronic Systems, headquartered in Mountain View, Calif., has more than 1,400 employees who provide total systems solutions for the military and national intelligence markets.

Download File