The Conundrums of the Costing World

1. The Conundrums of the Costing World Jairus Hihn October 28, 2008 SMAP Phoenix 23rd International Forum on COCOMO and Systems/Software Cost Modeling

2. Background The Jet Propulsion Laboratory is a Federally Funded Research & Development Center operated by the California Institute of Technology for the National Aeronautics and Space Administration. As part of the NASA team, JPL enables the nation to explore space for the benefit of humankind by developing robotic space missions to: • Explore our own and neighboring planetary systems. • Search for life beyond the Earth's confines. • Further our understanding of the origins and evolution of the universe and the laws that govern it. • Enable a virtual presence throughout the solar system using the Deep Space Network and evolving it to the Interplanetary Network of the future.

3. Size Growth & Stability Estimation & Planning Resources & Cost Schedule & Progress Product Quality MEsA Measurement Estimation & Analysis “Creating a Quantitative SW Management Culture” Helping Projects Measuring Development Performance • Project SW Measures Guide • Managing using Software Metrics Class • Measurement Support • Software Repositories • Cost • Defects • Foundation Measures • SW Engineering Models to support task planning • SW Cost Estimation Handbook • Software Cost Estimation Class • Estimation Support • Estimation Tools • Flight SW Cost Model (Team X) • SCAT: Probabilistic COCOMO • SLIC: Code Counter Organizational and Process Measures • JPL SW baselines and trends • Cost & Schedule Growth • Productivity • Reuse • Defect Density • Cycle Time • Measure Impact of SQI • Process Metrics • Benchmarking

4. Background (cont.) JPL has around 5,000 employees •  800 work with software in some capacity SQI has  15 FTE MEsA  4 FTE Assessed at Maturity Level 3 in September 2007 • Scope was Class B and C Mission Software

5. Cost Estimation Issues: Point 1 • All the people with power know just enough to be dangerous • Cost Estimators and Modelers must become change agents

6. Models Work But Getting Buy-in is Still a Struggle

7. OCM Key Points • Organizational Change does not come quickly or easily • The improvement process needs to be approached with many of the same deliberate methods and practices that are used in actual system development. • There are similarities between change at the individual and organizational level • The onus is on us to proactively reach out to customer/others and promulgate change • It is essential to proactively reach out to customers instead of merely waiting for them to come to you.

8. The Onus Is On Us • Think about it, what are the alternatives? • Blaming others • Why won’t they (colleagues) use our tool/methodology? • Why don’t those #$@@ managers force them to adopt our new approach? • Repeatedly OCM studies have documented that this approach never works • The Change Agent must accept responsibility forproactively reaching out to ‘customers’ and promulgating change

9. Percentage in Each Adopter Category Excerpted from http://www.valuebasedmanagement.net

10. We Are Here 7 6 Institutionalization Mechanisms to support sustaining the change 5 Mechanisms to support wider rollout of change Adoption Mechanisms to support measured success in piloting 4 Initial Use COMMITMENT TO CHANGE Understanding 3 Mechanisms to assure understanding Awareness 2 Mechanisms to promote awareness 1 TIME Adapted from Out from Dependency: Thriving as an Insurgent in a Sometimes Hostile Environment, SuZ Garcia and Chuck Myers, SEPG Conference, 2001 OCM Stages Internalization Contact

11. Parametric Cost Estimation Evolution

12. Cost Estimation Issues: Point 2 • Budget ‘bogies’ get set very early in lifecycle. Sometimes based on casual conversations. • You will typically get held to this number!! • Current proposal and planning process encourages/ demands under-estimating in early stages of lifecycle

13. What Should One Cost? What Flies Proposed Development Cost Performance Index In our environment we need to know what it will cost after all the *X!*! happens

14. Cost Estimation Issues: Point 3 • We improve our models but rarely do we improve our techniques • Our models have too many inputs • If samples are small not only is the data not normal it is not log normal either

15. Key Research Results Part 1 • Following is based on research with Tim Menzies using data mining techniques to analyze COCOMO • 2004-2008 • The results are … • Our models have too many inputs • Measures of RE go up with over specified models • Problem is what is best varies • Manual stratification does not lead to the ‘best’ model • E.g. a combination of flight SW and Class B ground produces a ‘better’ model then just selecting all your flight records and doing LC. • Nearest Neighbor searches for analogous records based on your current project model inputs

16. Key Research Results Part 2 • The results are … • The same approach is never best but some combination of the following always wins • Local Calibration • Column Pruning • Row Pruning based on Nearest Neighbor • Which is best is determined case-by-case • Recalibrating every time you estimate

17. Some Codicil Thoughts • There is no such thing as a point estimate that means anything, must account for uncertainty • We built SCAT a probabilistic COCOMO in Excel • Often the earliest simplest estimate is the best • Too often we talk ourselves into unrealistic low costs or get beaten into it. • Give Me Multiple high level estimates any day • Detailed estimates get lost in the trees and in an environment of requirements volatility it is a house of cards. • Whatever you do be consistent