The 6 Characteristics of Effective Metrics
I had the pleasure of meeting and working with Bob Lewis of Infoworld and CIO magazines recently. In looking through his many excellent blogs I noticed this one today. It sumarizes a viable metrics program very well and is compatible with the goal, question, metric approach. He calls it the 6 C's of effective metrics:
- "Connected to Organizational Goals: Good metrics are connected to important goals. In fact, they begin as important goals, stated in English.
- Consistent: Consistent metrics always go in one direction when the situation improves and the other direction when it deteriorates. If good doesn't always point in one direction and bad in the other, your metric will drive organizational dysfunction.
- Calibrated: Calibration means you get the same value in the same situation no matter who records it. It also means the data are free from sample bias and other quality problems.
- Complete: Anything you don't measure you don't get, so any useful system of measures must include all factors that are important to achieving the goal.
- Communicated: The purpose of metrics is to drive behavior. If you don't communicate their purpose, they won't drive behavior.
- Current: Goals change. Keep the old measures and you'll achieve your old goals, not your new ones."
Thank you for reading "Dan on Estimating", if you would like more information about Galorath's estimation models, please visit our contact page or call us at +1 310 414-3222.
FPGA Sticker Shock
Galorath’s Sam Sanchez works extensively with IC’s and with understanding the cost if IC’s. He provided the following:
Over the past year or so, we have been doing FPGA (Field Programmable Gate Array) material cost research updates in support of our next SEER-IC release and I can’t help being shocked at the recurring costs of some of the higher end FPGAs in the market. Once you pass the simple lower performance items, you start to see a large escalation in the per piece cost.
I remember years ago being surprised to see chips priced at $200 to $800. Now, I regularly see midsized state of the practice chips that go for $10K a piece (and I am not even talking about the leading edge 28nm stuff). At times, I come across several orders of magnitude above this. I guess it’s the sign of the times.
These devices can do more but they do not come cheap. Focusing only on performance, we might say who cares. However, as cost is becoming increasingly more critical, these new technology costs have to be examined closely. If for instance, you are targeting a portable device with a specific recurring cost bogey, you need to carefully consider what these new chip technologies mean not just in development costs but also in recurring costs. Depending on the size and amount of these FPGAs, their costs that dwarf all other BOM costs.
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.
2012 Defense Strategic Guidance Summary
The following is a summary by Galorath’s Chris Hutchings of the briefing on January 6, 2012 by Mr. Frank Kendall (Undersecretary of Defense for Acquisition, Technology, and Logistics) regarding the Strategic Guidance given by the President, the Chairman of the Joint Chiefs of Staff (Gen. Dempsey) and the Defense Secretary (Mr. Panetta).
- While the specifics on how the defense strategic guidance will affect the Pentagon’s budget are not yet known, it is intended that there will be a reduction in the region of $487 billion over the next ten years
- The US will continue to meet its global responsibilities, albeit in manner more aligned to the current and emergent requirements and constrained budgets. Specifically, non-priority capabilities including ‘protracted, large scale operations’ will be reduced while investments will be made in technologies that provide ‘an agile and decisive edge against all threats,’ including Cyber Warfare and Intelligence, Surveillance and Reconnaissance.
- Technology supremacy is seen as an integral part of a progressive strategy and, as such, those within the industrial base that develop and support such technology are central to the success of the strategy
- A significant emphasis was placed on a requirement for continued operational readiness of a smaller, leaner force that is easily regenerated and mobilized to meet the following primary aims –
- Strategic deterrence
- Defeat threats from terrorist organizations, including al Qaeda. He highlighted the use of Special Forces in this area.
- To react and defeat an immediate aggressor, while being able to ‘effectively inhibit the activities’ of a second aggressor
- In line with the ‘better buying’ initiatives of Deputy Secretary of Defense (Carter), the strategy will include mechanisms to incentivize industry to become more efficient
- Attention was drawn to two areas of specific risk to the success of the strategy, namely –
- Industry must be, and act as, a part of the solution to the challenges that the Pentagon faces
- Sequestration, a real threat to the aims of the strategy
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.
Cost Readiness Level (CRL) Establishing Cost Risk Based On Engineering Decisions
I am working on a paper for NASA’s project manager conference. In the course of this work I again reviewed the interesting work performed by Dr. Joe Hamaker while he was head of cost for NASA in establishing cost uncertainty before parametric models like SEER were used, and again after the system had been more finely modeled.
Like TRLs, CRLs are designed to communicate the quality of the product—its fitness for use. CRLs have been constructed to run the same 1 to 9 ordinal scale as TRLs, except we pay little attention to cost estimates at CRL 1-3. In the case of CRLs, it is the quality of the cost product itself—the estimate’s fitness for use –that we are measuring
Joe equated cost readiness levels to technology readiness levels (TRL) to cost readiness levels and came up with the following: Read more
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.
APMP Cost Task Force Industry / Government Questions & Responses
Here is the summary of the questions and responses from 2010 that were obtained regarding the APMP task force charter to make government and industry work better together in cost analysis. This is posted in advance of the task force telephone conversation on December 21, 2011.
If you would like an invite to that call or would like to join the task force please email Dan Galorath.
Probe further into causes for differences between Industry proposed costs and Government evaluated most probable costs
Understand differences in the approaches behind developing Industry cost proposals and performing Government cost evaluations
Identify contradictory assumptions
Industry
How do you determine the price you will propose?
What type of estimate do you prepare?
How far down in the WBS do you prepare it?
Government
When preparing your internal estimate, what is your technical baseline?
How detailed are your estimates, e.g., level of WBS? What are your estimating methodologies? What is your historical data?
Would you consider providing this information to industry as part of the solicitation request, especially if there is an incumbent?
Why discrepancies might exist
What might industry do to provide a realistic cost proposal?
What might government do to receive a realistic cost proposal
Industry
What types of assumptions do you make to support your estimate?
Do you clearly define these as assumptions?
Do you desire more information from the Government?
How much detail do you provide to substantiate your price? Do you prepare detailed justifications?
Government
How much detail is needed to substantiate bidders’ cost?
Is there a format that you might like to see this information in so that you can compare apples-to-apples?
Cost Proposal Substantiation
What might industry do to provide more consistent substantiation?
What might the Government ask for to ensure that it gets the level of detail it is seeking?
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.
New SEER Labor Rate Calculator: A Leap Forward in “Should Cost” & “Will Cost”
I saw a demo today of our new labor rate calculator. It takes in various labor rate drivers and computes a viable labor rate. It even evaluates the cost of equipment, electricity, floor space, insurance, etc.
This is a great step forward in the “should cost” and will cost for product manufacturing. Buying organizations can describe the problem and see what a fair labor rate for the region, country, machine, etc. Mixed currencies are supported as well.
The following is a small example of the kinds of information that can be specified In this case it is configured for manufacturing.
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.
Frank Vogelezang Pricing Vs Costing & Proof That IT Systems Can Be Estimated
Frank Vogelezang of Ordina‘s presentation at the Galorath conference in the Netherlands yielded numerous interesting and well prepared presentations by customers and partners. The presentation included here: Estimating & Pricing of Application Management covered the processes for estimating what the problem will cost and the separate processes for pricing what will be charged to the customer.
Estimation Vs Pricing
Estimation was described as an engineering discipline while pricing was described as a commercial discipline that leads to a structure optimized to win the deal with an offering that meets customer acceptance criteria.
SEER Validation of the COMPLETE SYSTEM
The part I found even more interesting was the SEER validation included within the paper. This validation compared the IT total system cost, including software development estimated by SEER-SEM, IT infrastructure (hardware, bandwidth, etc.) and IT services estimated by SEER-IT, and other (pass through) items estimated via manual processes. The results were good even before any calibration, and even better after they calibrated with some of their history.
One Size Fits All Models Versus SEER
Frank pointed out the risks of using simple cost estimating relationships for estimating and defined the reasons why they choose SEER including:
- The estimating model needs to have a breakdown structure that can accommodate different cost-drivers
- Requirements for a supporting tool: Ability to facilitate a breakdown with different cost-drivers Based on experience data
- Possibility of calibrating the data with own experience
- SEER for IT as the basis for the model Six types of knowledge bases (out of 12)
- SEER for Software for Application Development calculations
WBS In SEER IT Infrastructure, IT Services, Software Development
Results before calibration (at a 50%) probability were slightly higher than actuals. With calibration they were within a few percent. And again, this was the entire system estimate including software, hardware, services, and other costs.Be sure to check out Frank’s blog http://www.ThePriceofIT.blogspot.com for more insights from him on a variety of relevant topics.
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.
APMP Cost Taskforce November 2011: Helping Calm the Storm of Budget Cuts Via Cost Analysis
During the November 2011 meeting the APMP cost task force regrouped with Dan Galorath as the Chair. The cost task force’s mission is to help resolve the differences between government and contractor positions in cost. The PowerPoint including the history and new goals is here. Quintin Redmond also offered his systems engineering paper Why Affordability Is A Systems Engineering Metric to the group.
We determined to resurrect the earlier document covering government and contractor positions and complete it as a report.
The original statement of the issue follows:
- DIFFERENCES IN ESTIMATES: There are often large differences between an offeror’s proposed cost and the Government’s realistic or Most Probable Cost.
- GOVERNMENT & INDUSTRY: This task force will involve Government and Industry Cost experts to discuss/brainstorm possible reasons. (It may require discussing or mapping each others processes to determine where these disconnects may be.)
- IDENTIFY DISCONNECTS: Identify disconnects between offerors’ approaches in developing cost proposals and the Government’s approach to evaluating cost proposals (including performing realism analysis and developing Most Probable Cost)
- HELP RESOLVE DIFFERENCES DUE TO ANALYSIS DIFFERENCES: Minimize differences between the proposed cost and the Government’s Most Probable Cost that is due analysis approach differences (e.g., Government point estimate or a 50% probability approach vs. an offeror’s 10% best case approach.)
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.
CMMI and SEER Applicability: Estimation, Planning & Control Processes and SEER Integration
Galorath’s Ton Dekker presented this paper covering SEI’s CMMI and how SEER supports the processes for estimation and planning, monitoring and control, and how to make estimation part of the CMMI success criteria.
- Estimation & Planning
- Monitoring & Control
- Measurement & Analysis
- Quantitative Project Management
- Risk Management
Thank you for reading “Dan on Estimating”, if you would like more information about Galorath’s estimation models, please visit our contact page or call us at +1 310 414-3222.