Case Study
Improve Commercial-off-the-Shelf (COTS) Integration Estimates
Northrop Grumman, Mission Systems
Kathy Bradford, Systems Engineer and Lori Vaughan, Cost Engineer
Achieve remarkable results with new modeling methods Northrop Grumman Mission Systems is a leading global integrator of complex, mission-enabling systems and services for federal agencies engaged in defense and intelligence activities, as well as federal civilian organizations, state and local governments and commercial clients.
A Northrop Grumman Mission Systems project involved building the software system for a ground station. The purpose was to improve launch and control services of military satellites by increasing automation 
and simplifying launch operation. The system supported the Air Force's day-to-day mission of providing command and telemetry data via the Air Force Control Network.
Northrop Grumman Mission System's original proposal was based largely on internal research and development efforts undertaken at the company's own expense. The cost of the project was estimated using Galorath Incorporated's SEER-SEMTM (Software Estimation Model), a parametric tool that is a standard for software cost prediction and modeling at Northrop Grumman Mission Systems. SEER-SEM came with a database of thousands of completed projects from client-server, embedded, distributed, and stand-alone systems, as well as varied MIS applications, graphics, and signal processing environments.
The project was completed on schedule, met the customer's expectations, and declared a success. Upon completion, the work analysis was put through SEER-SEM and the results fell within seven percent of the actuals.
The job stood out because of its high COTS content. The COTS integration costs were not separately identified or estimated on the project so analysts wanted to see if they could get closer by dissecting the COTS elements. They decided to revisit the estimate to determine whether it could have been improved by a new method of estimating COTS integration costs that had been developed recently, but had not been used on an actual project.
Revisiting the estimate Northrop Grumman Mission Systems estimators began by establishing a one-page outline of the project that included major milestones such as the authorization to proceed, the DD250 form signifying the completion of the project, technical reviews, incremental delivery schedule, and formal test events. The outline also defined the functional and product organization in the project including contractors and subcontractors and all major elements of cost such as products and subsystems, sites, deliveries, and major COTS elements by subsystem.
The next step was analyzing the actual costs by mapping the project work breakdown structure (WBS) into the standard Northrop Grumman Mission Systems model called the Process Capability Database (PCDB). The PCDB is Northrop Grumman Mission Systems' central repository for historical data and includes work descriptions, labor, subcontractor, and other direct cost data
The actual costs were re-summarized using the standard WBS to determine the cost of each subsystem. The development effort was divided into hardware and software efforts; support efforts such as configuration management, testing, and quality assurance were also broken out. Time-phased actuals were used to establish headcount by milestone, discipline, and subsystem.
The next step involved generating a detailed count of the actual code. The configuration management system contained 58 different file types, which is not surprising since COTS integration systems typically 
have many file types. The contents of the project were differentiated based on file type extensions into 3GL and 4GL code, scripts, tables, and intermediates. The code created for this project was counted using source lines of code (SLOC), lines of executable code as defined by syntax and excluding comments, blanks, continuation lines, and pre-processor lines. The code was further broken down into the following categories:
1) New: new code developed from scratch;
2) Reuse: source code created elsewhere but used with some re-engineering and testing effort;
3) Derivative: multiple source files derived from one original with some modification; and
4) Non-product: code that was not part of final delivery such as test scripts and stubs.
Product development IPT leads were interviewed to determine the characteristics of the development environment and staff in order to establish the SEER-SEM input parameters (the original proposal is not provided for their reference).
Estimating COTS Integration Efforts
Summarizing the COTS integration efforts provided a greater challenge. The first step was sizing the COTS. Analysts spoke to vendors, but in many cases they didn't know or were not willing to divulge the size of their code. The SEER-SEM quick-sizing tool proved to be invaluable in these cases. The user entered whatever information was available into the tool to estimate COTS size.
In some cases, engineers made comparisons between the COTS programs used in the project to well-known products in the SEER-SEM data base. In others, they searched the COTS product documentation for well-known function names and inspected the index for known elements. Based on this information, the quick-sizing tool provided size counts that were later adjusted by the proportion of the product actually used in the project. The quick-sizing tool was also used to generate estimates of the integration effort based on its database of existing projects. This highly detailed information on the actual COTS products, along with the revised COTS integration efforts, was used to create a final estimate. When factoring a product into this estimate, it was forced to restart with a different COTS product 28 percent into the schedule and another product was severely constrained by staffing problems. It also considered that in the original proposal, the script and 4GL were assumed to be only 25 percent as hard as 3GL coding, so only one in four lines of code was used in the estimate. For the final estimate, the 4GL and script were corrected and all line counts were used. Finally, the SEER-SEM quicksize estimating feature was used to estimate the impact of COTS integration function counts and parameters from IPT interviews.
The results from this final estimation model were remarkable. Seven of the eight products were significantly closer to reality using the modified modeling methods. The overall effort and schedule predicted were within two percent of the program actuals. Based on these results, Northrop Grumman Mission Systems estimators plan to use this same approach to verify the process and productivity profiles of on-going COTS integration programs nearing delivery. The results will be used to better determine intermediate milestones and help quantify the effects of mid-term COTS fallout. Over the long run, these new methods should provide better insight into what parameters should be used to rack the status and estimate to completion future efforts with major dependencies on COTS products.
