I am working on a paper for NASA’s project manager conference.  In the course of this work I again reviewed the interesting work performed by Dr. Joe Hamaker while he was head of cost for NASA in establishing cost uncertainty before parametric models like SEER were used, and again after the system had been more finely modeled.

Like TRLs, CRLs are designed to communicate the quality of the product—its fitness for use. CRLs have been constructed to run the same 1 to 9 ordinal scale as TRLs, except we pay little attention to cost estimates at CRL 1-3.  In the case of CRLs, it is the quality of the cost product itself—the estimate’s fitness for use –that we are measuring

Joe equated cost readiness levels to technology readiness levels (TRL) to cost readiness levels and came up with the following:

Cost Readiness levels (CRL)

CRL 1, 2, 3: Not ready

CRL 4: Cost fit for very preliminary engineering decisions and very preliminary budget use (+/- 45%)

CRL 5: Cost fit for preliminary engineering decisions and preliminary budget use (+/-35%)

CRL 6: Cost fit for PDR engineering decisions and PDR budget use (+/- 25%)

CLR 7: Cost fit for firm engineering decisions and firm budget constraints (+/-15%)

CLR 8: Cost fit for very firm engineerin decisions and very firm budget commitments (+/-5%)

CLR 9:  Actual cost of completed project

Technology Readiness Level Definition

The goal of technology readiness level is to categorize hardware, software, or system concepts as to their maturity.  The following are definitions for TRL.  TRL 1 is barely a gleam in the engineering eye while TRL 9 is proven, operational technology.  The folllowing are definitions from NASA.

TRL 1 Basic principles observed and reported: Transition from scientific research to applied
research. Essential characteristics and behaviors of systems and architectures. Descriptive tools
are mathematical formulations or algorithms.
TRL 2 Technology concept and/or application formulated: Applied research. Theory and
scientific principles are focused on specific application area to define the concept. Characteristics
of the application are described. Analytical tools are developed for simulation or analysis of the
application.
TRL 3 Analytical and experimental critical function and/or characteristic proof-of-concept:
Proof of concept validation. Active Research and Development (R&D) is initiated with
analytical and laboratory studies. Demonstration of technical feasibility using breadboard or
brassboard implementations that are exercised with representative data.

TRL 4 Component/subsystem validation in laboratory environment: Standalone prototyping
implementation and test. Integration of technology elements. Experiments with full-scale
problems or data sets.

TRL 5 System/subsystem/component validation in relevant environment: Thorough testing
of prototyping in representative environment. Basic technology elements integrated with
reasonably realistic supporting elements. Prototyping implementations conform to target
environment and interfaces.

TRL 6 System/subsystem model or prototyping demonstration in a relevant end-to-end
environment (ground or space): Prototyping implementations on full-scale realistic problems.
Partially integrated with existing systems. Limited documentation available. Engineering
feasibility fully demonstrated in actual system application.

TRL 7 System prototyping demonstration in an operational environment
(ground or space): System prototyping demonstration in operational environment. System is at
or near scale of the operational system, with most functions available for demonstration and test.
Well integrated with collateral and ancillary systems. Limited documentation available.

TRL 8 Actual system completed and “mission qualified” through test and demonstration in
an operational environment (ground or space): End of system development. Fully integrated
with operational hardware and software systems. Most user documentation, training
documentation, and maintenance documentation completed. All functionality tested in simulated
and operational scenarios. Verification and Validation (V&V) completed.

TRL 9 Actual system “mission proven” through successful mission operations (ground or
space): Fully integrated with operational hardware/software systems. Actual system has been
thoroughly demonstrated and tested in its operational environment. All documentation
completed. Successful operational experience. Sustaining engineering support in place.

Part 2 of the CRL blog will be published soon.

Related posts:

1. CECIM: A Capability Improvement Model For Cost Engineering Being a major fan of repeatable process for cost and...
2. Oh Glorious Day.. Weapons System Decisions To Be Made Based On Total Life Cycle Costs The October 13 2008 issue of Federal Times has a...
3. NPD 1000.5 NASA Cost-Risk Directive Just about hot off the presses (January 2009) is NASA’s...

I saw a demo today of our new labor rate calculator.  It takes in various labor rate drivers and computes a viable labor rate.   It even evaluates the cost of equipment, electricity, floor space, insurance, etc.

This is a great step forward in the “should cost” and will cost for product manufacturing.  Buying organizations can describe the problem and see what a fair labor rate for the region, country, machine, etc.  Mixed currencies are supported as well.

The following is a small example of the kinds of information that can be specified  In this case it is configured for manufacturing.

Related posts:

1. Coffee Maker Cost Estimation Using SEER for Manufacturing The department of Industrial Management & Systems Engineering at West...
2. SEER Estimate By Comparison Professional Released To All Users                 The SEER...
3. Estimating Best Practices The following are the bullet points from Dan’s paper on...

Frank Vogelezang of Ordina‘s presentation at the Galorath conference in the Netherlands yielded numerous interesting and well prepared presentations by customers and partners.  The presentation included here:  Estimating & Pricing of Application Management covered the processes for estimating what the problem will cost and the separate processes for pricing what will be charged to the customer.

Estimation Vs Pricing

Estimation was described as an engineering discipline while pricing was described as a commercial discipline that leads to a structure optimized to win the deal with an offering that meets customer acceptance criteria.

SEER Validation of the COMPLETE SYSTEM

The part I found even more interesting was the SEER validation included within the paper.  This validation compared the IT total system cost, including software development estimated by SEER-SEM, IT infrastructure (hardware, bandwidth, etc.) and IT services estimated by SEER-IT, and other (pass through) items estimated via manual processes.  The results were good even before any calibration, and even better after they calibrated with some of their history.

One Size Fits All Models Versus SEER

Frank pointed out the risks of using simple cost estimating relationships for estimating and defined the reasons why they choose SEER including:

• The estimating model needs to have a breakdown structure that can accommodate different cost-drivers
• Requirements for a supporting tool: Ability to facilitate a breakdown with different cost-drivers Based on experience data
• Possibility of calibrating the data with own experience
• SEER for IT as the basis for the model Six types of knowledge bases (out of 12)
• SEER for Software for Application Development calculations

WBS In SEER IT Infrastructure, IT Services, Software Development

 

Results before calibration (at a 50%) probability were slightly higher than actuals.  With calibration they were within a few percent.  And again, this was the entire system estimate including software, hardware, services, and other costs.Be sure to check out Frank’s blog http://www.ThePriceofIT.blogspot.com for more insights from him on a variety of relevant topics.

Related posts:

1. A Good Cost Estimate: The Foundation for the Art & Science of Pricing I thought this was an interesting article “A Good Cost...
2. The Total Ownership Cost of IT Systems It is interesting to see how the much of the...
3. Estimating Systems Engineering…. The Upcoming Book.. and SEER I had the opportunity to review Dr. Ricardo Valerdi’s (MIT)...

Here are the slides from the Galorath / David Consulting Group estimating clinic  held in New York City. There were interesting discussions encompassing dealing with impossible demands, understanding risk and risk management, sizing, and a number of other topics.  Note: The Galorath slides and the DCG slides are combined in the one PDF file.

Concepts of IT providing business value to the organization were also discussed in detail.

Thanks to all who attended and especially to Mike Harris and David Herren for their insights.

Related posts:

1. Galorath / Reifer Team To Provide Large Project Success Via Risk Assessments We are excited to announce a new addition to the...
2. Another View of Project Success From Dr. Dobbs While the Standish Group’s surveys are the most widely known...
3. The Business Value Of IT Book I opened my copy of this book, The Business Value of...

People are the most important asset in an engineering environment.  SEER models the productivity of teams well.

And drilling down,  there is a way to look at individual productivity based on how “into” their job they are.  People that are disengaged have about 45% less individual productivity than those that are nominally engaged in the job.  Worse yet, only about 1 in 10 disengaged people can ever get the the engaged level.

It is possible to improve the level of engagement with some employees, some of the time.  We will be briefing more detail on this at our New York event this week and and plan on offering help in evaluating employee engagement as well as what can be done to improve engagement in the near future.

Related posts:

1. Software Staff Size Still Impacts Productivity: Brooks Law Lives! When drawing conclusions from data it is important to analyse...
2. List of Documents On Software Productivity This is a work in process list of documents covering...
3. Using the SEER Effective Technology for Productivity Assessment and Benchmarking While I prefer to use productivity (which consolidates everything about the...

Here are some of the questions that were answered after the webinar with Bill Curtis, David Herron, and Dan Galorath.  They were answered on Cast software’s Facebook page.

What about aerospace component level software and how would you apply a cost factor their  complexity factors?

What is IFPUG Back Fired Points, how does it help?

Daniel Galorath Backfiring  means counting lines of code then using a number of lines per function point to approximate function points.  I think it is much better than nothing for a finger in the wind.  Many others in the industry object strongly to it.

Another question – What type of information is needed to start using a software analysis and measurement tool?

Bill Curtis First  you need to know how you want to use the results.  That will help you establish criteria for evaluating which of the various tools best meets your needs.  The Goal-Question-Metric paradigm is a good guide for determining what your measurement needs are.

Here’s  another question from the webinar – Would like to understand how we can  improve our ability to capture metrics (defect density either using effective loc and or functional point) for applications that rely heavily on database’s where the logic resides – e.g to map biz rules existing in table rows/columns etc?

Daniel Galorath Generally  you would count the work involved in building the database, not the entry of the data into the database itself. SEER will provide estimates of defect density, etc.  From my understanding CAST can provide measurement of this.

What  is the panel’s opinion on LLOC to FP translation such as QSM language by language table which they developed from their database.

Do you have to have a technology inventory?

How do you extend the IT governance to software suppliers? what requirements to pose to them?

Bill Curtis Customers are starting to write measurable quality targets such as robustness or security targets into their outsourcing contracts as the equivalent of service level agreements. They then establish a Quality Gate where all software received is measured and evaluated before being put in operation. If the supplier’s software falls below the quality target they must remediate the code or face a financial penalty.

Related posts:

1. The Future of Software Analysis Measurement Webinar Featuring Bill Curtis, David Herron and Dan Galorath We have some exciting events coming up this month with...
2. Estimation Plus Process and Measurement Rigor Yields More Successful Projects: SEER and Tracer It has been rewarding working with the Computer Aid  (CAI)...
3. Clean Energy Analysis Software The RETScreen Clean Energy Project Analysis Software (from Natural Resources...

We have some exciting events coming up this month with the first: The Future of Software Analysis and Measurement on October 12, 2011. I am really excited to share the virtual podium with Bill Curtis, Senior VP and Chief Scientist of CAST Corporation and with David Herron, David Consulting Groups VP of Knowledge Solution Services as moderator.

Bill and I both spoke at a software engineering conference last year and I got very excited about his work in software analysis with CAST. Besides Bill being an engaging speaker his content was very illuminating, covering issues of existing software, its complexity and reliability. He ev!–more–
en showed the number of latent defects in software of various languages. I have greatly summarized some of the a href=http://www.galorath.com/wp/software-defects-in-fielded-software-cast-analysis.phpsoftware defect conclusions /aelsewhere on this BLOG and CAST has been instrumental in recognizing and quantifying the a href=http://www.galorath.com/wp/500-billion-it-debt-for-deferred-maintenance.phptechnical debt/a

And with David Herron, one of the most knowledgeable people in the measurement community, this should be a do not miss event.

Details of the event follow as does a link to signup. Hope you can make it.
blockquotestrongThe Future of Software Analysis and Measurement/strong

October 12, 2011 8:00am Pacific, 11:00am Eastern, 4:00pm London

a href=http://www.castsoftware.com/news-events/event/future-of-sam?gad=glrClick this Webinar Link to sign up/a

Join us on October 12th to hear from an exciting lineup of experts on the Future of Software Analysis and Measurement: Dan Galorath, President CEO of Galorath Inc and Bill Curtis, SVP Chief Scientist, CAST will have an engaging discussion moderated by David Herron, VP, Knowledge Solution Services, David Consulting Group.

These industry veterans will share experiences with their client’s software development processes and discuss how Software Analysis and Measurement tools coupled with Parametric Estimation models can impact organizational performance through increased ROI, customer satisfaction and business value.

The panel will provide insightful and actionable steps that will make an immediate impact on your strategy including how to:

• Drive organization value by fueling Estimate and Measurement practices within an enterprise

• Build the funding rationale through proven economic impact models

• Establish the ROI from Estimate and Measurement practices and process/blockquote

Related posts:

1. The Future of Software Analysis and Measurement : Expert Panel Questions & Answers Here are some of the questions that were answered after...
2. Measurement and Providing Value to the Business ISMA Keynote 2011 Here is a copy of my keynote talk at the...
3. Issues in Software Measurement & Estimation ISMA 2010 Measurement is a wonderful thing.  However measurement without standards and...

IBM’s Murray Canter published an interesting article in the communications of the ACM covering calculating and improving the ROI in software systems.   Murray shows how to compute the “investment value” of incomplete software and illustrates why it does have value showing how to compute the net present value and the return on investment of this in process work, using Monte Carlo simulation.  Murray states two axioms:

• Costs and benefits occur over time, so their present values are found through NPV equations
• The future values of costs and benefits are random variables, described as a statistical distribution

I should note this requires a subscription to ACM digital content.

Its abstract states:

“Constrained by limited budgets, most enterprises find it essential to apply unprecedented business discipline to the business function of software and system delivery (SSD) across entire software and system life cycles. For this reason, the CIO, CTO, or VP of software or systems development may be under increased scrutiny from the corporate chief finance office (CFO). When conversing with the CFO, money talks, so only one of two sorts of conversations can take place: software and systems as cost center or software and systems as value-creation center.”

Related posts:

1. Computing the Business Value of IT While I am a strong proponent of examining the business...
2. Software as a Service vs. Service Oriented Architecture vs. Cloud Computing I have seen a lot of confusion as to the...
3. The Cost of Cloud in the Sky Computing Part 1 Looking at ways to increase IT infrastructure without worrying about...

Here is a copy of my keynote talk at the 2011 ISMA measurement conference, Measurement and Management and Business Value.  The real point is the IT and measurement personnel have the information, knowledge and skill set to provide value to the business, far beyond the costs generate.  But the information needs to be communicated in terms leaders can understand rather than the techie language we all think and speak in… AND if IT starts showing how it contributes to the business and becomes a profit center instead of a cost center IT will get more money to apply to more valuable things.

UPDATE: Someone also asked for a reference communicating the language of management.  I believe this publication on business case analysis to be helpful in that regard.

PS In my talk I referred to software and measurement people as geeks.  To many the term geek is a source of pride.  One person in the talk was offended by being called a geek.  My sincere apologies.  However modern vernacular often defines a geek as a lover of technology, a software developer or someone with an intense love of mathematics.  I consider myself a geek.  And my friend Paul Glen makes his living by “leading geeks.”

Related posts:

1. Galorath Keynote ISPA March 2011 Today my conference presentation involved understanding and estimating value in...
2. Issues in Software Measurement & Estimation ISMA 2010 Measurement is a wonderful thing.  However measurement without standards and...
3. Communicating with Management Via Measurement This is the abstract for my keynot talk at the...

I stumbled across a working draft of an excellent document on my hard drive, produced, I believe, by a gentleman from Texas Instruments some years ago.  I thought this list of why we estimate cost and schedule was excellent and still relevant.

Why We Estimate Schedule & Cost

• To scope proposed tasks
• To explore alternative system concepts

design to cost/budget

• To explore alternative design concepts
• To explore alternative proposals for enhancements and upgrades
• To identify key design elements
• To identify key process parameters

prioritize needs vs wants

• To identify key assumptions
• To identify and quantify uncertainties
• To identify tasks and their relationships
• To assess schedule feasibility
• To identify, allocate and schedule resources
• To assess an organization’s ability to perform within targeted costs
• To evaluate the consequences of internal and external constraints
• To establish achievable objectives
• To establish a basis for quality service
• To establish commitments
• To bound the risk against customer needs
• To balance levels of risk against customer needs
• To provide a basis of successful risk management

build vs buy analysis

• To prepare successful proposals
• To evaluate proposals from competing bidders
• To establish baselines for project tracking

enhance/reuse vs redesign analysis

• To predict life-cycle costs
• To predict returns on investments
• To provide information for establishing business and investment strategies

Related posts:

1. Step One: Establish Estimate Scope and Purpose Establish Estimate Scope and Purpose Define and document expectations. When...
2. Step Seven: Estimate Validation and Review Step Seven: Estimate Validation and Review At this point in...
3. Design To Cost Metrics Can Help Increase Project Value While Reducing Cost An interesting paperBy Quentin Redman and George Stratton in the...