Proxy Estimation Costing for Systems (PECS)

1. Proxy Estimation Costing for Systems (PECS) October 2012 Reggie Cole Lockheed Martin Senior Fellow

2. Discussion Topics • Why Do We Need Yet Another Cost Model? • The gap in early-stage system cost modeling • Systems Engineering Effort as a Proxy Estimator for System Cost • And the role of COSYSMO is arriving at this proxy estimate • Proxy Estimation Costing for Systems (PECS) • Derivation of the PECS Model • The PECS modeling approach • Case Study for Affordability Analysis Using the PECS Model • The real power of the PECS model

3. Cost Modeling Needs Change Over Time in Terms of Speed and Accuracy – So Does Solution Information We Have a Good Selection of Tools for Late-Stage Cost Modeling Detailed Solution Description Cost Estimate ± 5% High-Level Solution Description Cost Estimate ± 10% Increasingly Refined Information About the Solution Increasingly Refined Cost Estimate Problem-Space Description Increasingly Refined Solution High-Level Solution Assumptions Cost Estimate ± 20% Cost Estimate ± 25% We Have Gaps in Early-Stage Cost Modeling Increasing Effort and Cost-Modeling Expertise

4. Systems Engineering Effort as a Proxy Measure of Overall System Size and Complexity • Proxy Measures • Proxy measures are used when you cannot directly measure what you want to measure – and when an indirect measure provides sufficient insight • Proxy measures are often used in clinical studies since direct measurement is often infeasible or can even alter the outcome • It is not always possible to directly measure what you want to measure – or directly estimate what you want to estimate • System Engineering Effort is a Proxy Measure for System Cost • There is strong evidence for the link between systems engineering effort and program cost – dating back to a NASA study in the 1980s • The optimal relationship between systems engineering effort and overall program cost is 10% - 15% • Industry has long used a parametric relationship between software cost and systems engineering cost for software-intensive systems • Systems engineering effort can be an effective proxy measure for overall system cost H. Dickinson, S. Hrisos, M. Eccles, J. Francis, M. Johnston, Statistical Considerations in a Systematic Review of Proxy Measures of Clinical Behaviour, Implementation Science, 2010 E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004

5. COSYSMO 2.0 Model Parameters Provide a Rich Assessment of System Size, Complexity and Reuse Size Drivers Number of System Requirements Initial Estimate of System Size Number of Major System Interfaces Number of Critical Algorithms Number of Operational Scenarios Requirements Understanding Reuse Factors Cost Drivers Architecture Understanding Managed Elements Level of Service Requirements Adopted Elements Migration Complexity Scaled Estimate of System Size Deleted Elements Technology Risk Modified Elements Level of Documentation Required New Elements Diversity of Installed Platforms Consolidated Cost Driver Factor Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Estimate of Systems Engineering Effort…Also a Biased Proxy Estimator for System Scope…And System Cost Multisite Coordination Level of Tool Support

6. An Approach for De-Biasing the Proxy Estimator –Relationship Between SE Effort and Total Effort NASA data supports a 10%-15% optimal allocation of systems engineering effort as a portion of overall program effort INCOSE study on the value of systems engineering also supports a 10%-15% optimal allocation of systems engineering as a portion of overall program effort W. Gruhl, Lessons Learned, Cost/Schedule Assessment Guide,” Internal Presentation, NASA Comptroller’s Office, 1992 E. Honour, “Understanding the Value of Systems Engineering,” INCOSE, 2004

7. The PECS Cost Function This Model is Well Positioned for Monte Carlo Analysis

8. The PECS Model – Putting It All Together Estimator Bias Function is Based on the Well-Established Relationship Between SE Effort and Overall Program Effort Proxy Estimation Costing for Systems (PECS) • Size Drivers (Problem Space) • Customer Requirements • System Interfaces • Major Algorithms • Operational Scenarios • Complexity Drivers (Problem/Solution) • Requirements Understanding • Architecture Understanding • Level of Service Requirements • Migration Complexity • Technology Risk • Documentation Needs • Installations/Platform Diversity • Levels of Recursion in the Design • Stakeholder Team Cohesion • Personnel/Team Capability • Personnel Experience/Continuity • Process Capability • Multisite Coordination • Tool Support • Reuse Factors (Solution Space) • New • Modified • Deleted • Adopted • Managed SE Effort is an estimator for total system cost…but it is a biased estimator Estimator De-Biasing Monte Carlo Analysis of System Cost Three different COSYMO scenarios – optimistic, expected & pessimistic – provide the basis for the Monte Carlo analysis of system cost

9. Case Study – The COSYSMO Scenarios • The case study is based on a large C2 system. Initially developed 20 years ago, the system was unprecedented. Twenty years later, a replacement system is needed. While the initial development was unprecedented, the replacement system is not, which drives down the size drivers (through reuse) and cost drivers. • The case study looks at three cost scenarios: • Case 1 – The original unprecedented system (for calibration purposes) • Case 2 – Replacement system (as a new development) • Case 3 – Replacement system (as a largely COTS/GOTS approach) COSYSMO Scenarios for PECS – Three Scenarios for Each Case

10. Case Study – The Monte Carlo Analysis Case 1 Average 80/20 Cost = $1.9B Used as a calibration point for the model Case 2 Average 80/20 Cost = $77M Initial Solution for Replacement System Case 3 Average 80/20 Cost = $30M More Affordable Solution, Based on COTS/GOTS Solution The PECS Model Enables Rapid Turn-Around Analysis of Alternatives and “Should Cost” Analysis

11. Conclusion • The PECS Model is Based on Well-Established Approaches • COSYSMO provides the basis for estimation of systems engineering effort – and a biased proxy estimator for overall system cost • There is a well-established relationship between systems engineering effort and overall effort used to de-bias the COSYSMO-modeled effort • Monte Carlo analysis is a well-established technique for cost modeling • The PECS Model Can Improve System Cost Modeling • The PECS Model occupies an important niche – fully parametric system cost modeling in the early stages of system definition • The PECS Model can serve as a powerful affordability analysis tool – supporting rapid-turnaround analysis of alternatives • But…the PECS Model is not a replacement for existing models • Next Steps • Broader validation of the model • Cross-industry review of the model