Richard Hartley Deputy Assistant Secretary of the Air Force Cost and Economics 28 Jan 2010

1. Richard Hartley Deputy Assistant Secretary of the Air Force (Cost and Economics) 28 Jan 2010

3. PerformanceCost Growth Analysis  Differences from previous chart Data from Dec 2006 SAR, vice Dec 2002 SAR Differences from top table to bottom table Top Table There are 30 completed AF programs in the database, and 65 completed DOD programs in the database Bottom table Represents ongoing programs that meet 5 years past MSB requirement 7 additional AF programs can now be analyzed because they have reached the 5 years past MSB requirement AEHF C-130 AMP C-5 RERP Global Hawk JSF (F-35) NPOESS Wideband Gapfiller Satellites Note that �AF 5 years past MSB� looks worse for ongoing programs �AF growth to date� is significantly higher than �AF at completion� AND is higher than �Projected growth at completion� calculation �Projected growth at completion� is now an average of the historical growth factor for completed programs and the current (incomplete) growth factor (growth to date / growth at 5 years post MSB) for ongoing programs This average will likely under-represent actual growth because I am forced to use the incomplete growth factor for current programs 8 ongoing AF programs started prior to 2000 include: EELV F-22 Global Broadcast Service GPS BLK2R JASSM JPATS NAS SBIRS-HighDifferences from previous chart Data from Dec 2006 SAR, vice Dec 2002 SAR Differences from top table to bottom table Top Table There are 30 completed AF programs in the database, and 65 completed DOD programs in the database Bottom table Represents ongoing programs that meet 5 years past MSB requirement 7 additional AF programs can now be analyzed because they have reached the 5 years past MSB requirement AEHF C-130 AMP C-5 RERP Global Hawk JSF (F-35) NPOESS Wideband Gapfiller Satellites Note that �AF 5 years past MSB� looks worse for ongoing programs �AF growth to date� is significantly higher than �AF at completion� AND is higher than �Projected growth at completion� calculation �Projected growth at completion� is now an average of the historical growth factor for completed programs and the current (incomplete) growth factor (growth to date / growth at 5 years post MSB) for ongoing programs This average will likely under-represent actual growth because I am forced to use the incomplete growth factor for current programs 8 ongoing AF programs started prior to 2000 include: EELV F-22 Global Broadcast Service GPS BLK2R JASSM JPATS NAS SBIRS-High

4. A Different Look Dollarized Growth Trends by Decade To project cost at completion in 00s: Applied average of historical growth factor (growth at completion / growth at 5 years post MSB) for completed programs and current growth factor (growth to date / growth at 5 years post MSB) for ongoing programs 70s programs A-10 AGM-65D ALCM AWACS B-1A DSCS-III E-4 AABNCP EF-111A F-15 F-16 GLCM GPS-BLK1&2 80s programs AIM-120A (AMRAAM) ATARS B-1B C-17 CBU-97B (SFW) CELV (Titan IV) E-3A (AWACS) RSIP E-8A (JSTARS) JTIDS LANTIRN OTH-B To project cost at completion in 00s: Applied average of historical growth factor (growth at completion / growth at 5 years post MSB) for completed programs and current growth factor (growth to date / growth at 5 years post MSB) for ongoing programs 70s programs A-10 AGM-65D ALCM AWACS B-1A DSCS-III E-4 AABNCP EF-111A F-15 F-16 GLCM GPS-BLK1&2 80s programs AIM-120A (AMRAAM) ATARS B-1B C-17 CBU-97B (SFW) CELV (Titan IV) E-3A (AWACS) RSIP E-8A (JSTARS) JTIDS LANTIRN OTH-B

5. Sources of Cost Growth SAR DATA - Economic = .3/Engineering = 18.6 = 18.9/56.5 = 33.45 - Estimating = 30/56.5 = 53% - Quantity = 2.6/56.5 = .46 - Schedule = 4.5/56.5 = .79 - Misc = support/other RAND DATA - Economic = Cost Estimates = 18+Schedule Est=1+Tech issues=.6 = 19.6/56.5 = 34 - Estimating = Requirements=17.5/56.5 = 31 - Quantity = 4.3/56.5 = 7.6 - Schedule = 6.0/56.5 = 10.6 - Miscellaneous = [5.2+1.0+4.8+(-1.9)]/56.5 = 16.1 External events - ( e.g. Challenger disaster on Titan pgm) Inter/intra pgm funding transfers � Color of money transfer within a pgm (between development and procurement or O&M) or between programs Affordability � Decision by OSD, Congress, or the service to change the pgm because of the cost issues Financial Exchange rate � Difference between predicted & actual exchange rates Inflation � Difference between predicted and actual inflation SAR DATA - Economic = .3/Engineering = 18.6 = 18.9/56.5 = 33.45 - Estimating = 30/56.5 = 53% - Quantity = 2.6/56.5 = .46 - Schedule = 4.5/56.5 = .79 - Misc = support/other RAND DATA - Economic = Cost Estimates = 18+Schedule Est=1+Tech issues=.6 = 19.6/56.5 = 34 - Estimating = Requirements=17.5/56.5 = 31 - Quantity = 4.3/56.5 = 7.6 - Schedule = 6.0/56.5 = 10.6 - Miscellaneous = [5.2+1.0+4.8+(-1.9)]/56.5 = 16.1 External events - ( e.g. Challenger disaster on Titan pgm) Inter/intra pgm funding transfers � Color of money transfer within a pgm (between development and procurement or O&M) or between programs Affordability � Decision by OSD, Congress, or the service to change the pgm because of the cost issues Financial Exchange rate � Difference between predicted & actual exchange rates Inflation � Difference between predicted and actual inflation

6. 6 Observations Current trends not favorable Projected growth at completion >2x prior generations Dollarized cost growth trends cause for concern Historically, most growth occurs prior to �Milestone B + 5 years� point, recent programs displaying growth well beyond Cost growth statistics are lagging indicators Start to measure at 5 years past Milestone B Today�s stats reflect decisions of mid-90�s, early 00�s Recent initiatives help, but more action necessary

7. Why is it So Hard? TECHNICAL/SCHEDULE BASELINE Optimism - Influence from industry, influence on industry � pressure from competition, budget, senior management; natural biases � cognitive, organizational Resources/experience; accountability; independence � databases Identification of risk PROGRAMMING/BUDGETING Ability to budget to cost estimate Delta between gvmt estimate and contractor estimate Ability to hold MR/risk, budget taxes, etc. Budgeting realistic cost estimates �No bill before its time,� optimism, competing influences Budgets established/altered w/o benefit of sound cost estimate Budget stability CONTRACTING Contracting/ cost estimating disconnect -- cost realism vs. negotiating best value Resources/experience, skills, databases PROGRAM MANAGEMENT Optimism (Same as Tech/Sched b/line); Resources/experience, accountability REQUIREMENTS COMMUNITY Realism, sufficient interaction/iterations with cost, PM, technical community SENIOR MANAGEMENT Optimism (Same as Tech/Sched b/line); Inconsistent value placed on "alternative view" of cost/technical/schedule issues; Resources/experience, accountability INDUSTRY Optimism (Same as Tech/Sched b/line); Pressure to win (profit vs. business base), pressure from government to meet unrealistic goals, negotiation process, cost estimating capability (similar to gvmt), �gaming� common cost models� POLITICAL ENVIRONMENT Competing priorities often lead to budgetary "hits" to programs, leads to less efficient development and procurement schedules - higher costs TECHNICAL/SCHEDULE BASELINE Optimism - Influence from industry, influence on industry � pressure from competition, budget, senior management; natural biases � cognitive, organizational Resources/experience; accountability; independence � databases Identification of risk PROGRAMMING/BUDGETING Ability to budget to cost estimate Delta between gvmt estimate and contractor estimate Ability to hold MR/risk, budget taxes, etc. Budgeting realistic cost estimates �No bill before its time,� optimism, competing influences Budgets established/altered w/o benefit of sound cost estimate Budget stability CONTRACTING Contracting/ cost estimating disconnect -- cost realism vs. negotiating best value Resources/experience, skills, databases PROGRAM MANAGEMENT Optimism (Same as Tech/Sched b/line); Resources/experience, accountability REQUIREMENTS COMMUNITY Realism, sufficient interaction/iterations with cost, PM, technical community SENIOR MANAGEMENT Optimism (Same as Tech/Sched b/line); Inconsistent value placed on "alternative view" of cost/technical/schedule issues; Resources/experience, accountability INDUSTRY Optimism (Same as Tech/Sched b/line); Pressure to win (profit vs. business base), pressure from government to meet unrealistic goals, negotiation process, cost estimating capability (similar to gvmt), �gaming� common cost models� POLITICAL ENVIRONMENT Competing priorities often lead to budgetary "hits" to programs, leads to less efficient development and procurement schedules - higher costs

8. 8 Delusions of Success � How Optimism Undermines Executives Decisions Harvard Business Review, Dan Lavallo and Daniel Kahnemon When planning major initiatives Routinely exaggerate benefits and discount costs when projecting risky project Sets up project for failure; psychologists �planning fallacy� -- spin success scenarios and overlook potential for mistakes Optimism traced to cognitive biases and organizational pressures People highly optimistic most of the time � exaggerate talents and degree of control; attribute negative consequences to external factors Competition ($, time) for new projects � incentive to accentuate the positive Anchoring magnifies optimism; initial plan accentuates positive, skews subsequent analysis Most pronounced for initiatives companies have never attempted before Temper with �outside view� Supplements traditional forecasting w/ statistical analysis of analogous efforts � reality check on initiative inside view Don�t remove optimism, but temper effects � balance between optimism and realism - goals (motivate) and forecasts (decide whether to make commitment in the first place) Optimism

9. Congressional Direction FY05 HASC Strategic Forces Subcommittee H.R. 109-89, NDAA 2006 � �Cost analysis for space acquisitions� P.L. 109-163, NDAA 2006 Section 801 � Requirement for Certification Before Major Defense Acquisition Program May Proceed to Milestone B. Section 802 � Requirements Applicable to Major Defense Acquisition Programs Exceeding Baseline Costs Section 804 � Reports on significant increases in program acquisition unit cost or procurement unit cost of Major Defense Acquisition Programs H.R. 109-452, NDAA 2007 � �Critical shortage of acquisition professionals needs a human capital strategic plan� 9

10. Congressional Direction(continued) P.L. 109-364, NDAA 2007 � �Provisions for MDAPs� Section 816 � Major Automated Information System Programs Section 820 � Government Performance of Critical Acquisition Functions Section 853 � Program Manager Empowerment and Accountability P.L. 110-181, NDAA 2008 Section 852 � Dept of Defense Acquisition Workforce Development Fund P.L. 111-23, 2009 Weapons System Acquisition Reform Act (WSARA) Director, Cost Assessment and Program Evaluation (CAPE) Tightens provisions for troubled programs to continue More for JROC to validate programs Director, Def Research and Engineering annually assess tech maturity and integration risk on MDAPs GAO evaluate accuracy of O&M cost estimates Organizational conflicts of interest; cash awards for acquisition personnel 10

11. WSARAWhat it Means to You CAPE concurrence on cost estimates for POM, 2366 certifications (MS A & B) Full funding requirements More robust MS A cost estimates Documentation required, stored in repository

12. AF FM Acquisition Excellence 12

13. Improving Cost Estimating 13

14. PPBES Decision Support(XXXX) Portfolio

15. Program Control, Budgeting& Earned Value Management 15

16. Strategic Initiatives Implement WSARA Integrate w/ AIP Leverage Sec. 852 Workforce Development Fund People, training and education, infrastructure, retention Rebuild APDP (BUS-CE, BUS-FM) Strengthen ties w/ engineering, PMA&E and contracting Independent technical assessment (ITA), Independent schedule assessments (ISA), IBRs, negotiations, EVM Identify, initiate, and execute FM-level initiatives FM DT attention (career paths, training, education and development); program control and EVM; AFIT; military and civilian workforce

17. Air Force Acquisition Improvement Plan (AIP) Revitalize the AF acquisition workforce Increase the workforce by over 2,000 employees (247 officers, 11 enlisted, and 1,804 civilian) All new military added in FY10 and civilian between FY10 and FY13 Improve requirements generation process Require incremental acquisition strategies that reduce cost, schedule and technical risk Instill budget and financial discipline Program baselines must be based on realistic schedule and technical assumptions and accurate cost estimates �not just the cost of the lowest bid Improve AF major systems source selections Must ensure our personnel have the required experience and training to conduct source selections Establish clear lines of authority and accountability within acquisition organizations Current wing structure appears to have created a number of negative unintended consequences 17

18. Tactical Initiatives

19. Way Ahead 19

20. 20

21. 21 Questions

22. Annual POEs and NACAs IAW standards and guidelines Structured for PMs, PEOs, SAF/AQ, SAF/FM Support AFCS, SAF/FM, SECAF and CSAF Cost conscious/informed panel, group, board deliberations Reliable, realistic cost estimating Portfolio risk analysis Feeds MDAP Issue Team 2366 certification, AT&L/CAPE requirements, WSARA documentation (AFI)

23. Closing Perspective King For a Day Increase number of cost estimators Rebuild AF cost career field Create cost estimator job series Require standard WBS and cost/technical CDRL Facilitate maximum objectivity through org structure Establish �institutional� independent technical assessment capability

24. Require �auditable� performance metrics Improve source selection cost proposals Increase infrastructure investment Establish/fund gov�t/industry cost IPTs Improve cost risk analysis, (generally) budget to mean Improve cost/contracting interface

25. Strategic Level 25

26. Tactical Level 26

27. Acquisition Workforce Growth

28. Acquisition Additions 28

29.  80% Confidence LevelA Wise Consideration, But Be Very Careful 

30. BUS-CE Certification Program l have been doubled from 1,2, and 4 to 2, 4, and 8. This was done to place value on experience. It almost takes 4 years to be exposed and fully understand the cost analysis of a program. We have kept ACQ 101 and ACQ 201---colored in light blue BCF 102 and 103 will be streamlined to merge into 1XX---the same is true for BCF 205 and 211, which will be reviewed to extract core material for the BCF 2XX. Both 1XX and 2XX are our BCEFM Common courses colored in yellow/purple The courses in blue are courses that would be required at each level�those in red lettering are completely new courses. Note for Level I, we have BCF 106 (on-line and BCF 107 (resident)�this will replace the current BCF 101. For level II, the re are some modifications for 206 (Cost Risk Analysis) and 204 (intermediate cost) as well as the introduction of 4 CLMs, which are on-line short courses that could be taken in approximately 4 hours each. At level III, there is BCF 3XX, which will cover advance topics, and streamlined topics from the current BCF 301. Note also that the courses are heavily offered in the first two levels---this is done on purpose---in order to do cost estimating, you need to know a lot about a lot of different topics, we wanted to address the foundational and applicable knowledge within the first 4 years, then allow the individual to put into practice their knowledge and then come back by Level III with �total� understanding of cost and review and elaborate on cost advance topics. One final element of this program is that we want to allow each individual to select course from the �plus� part of the Cost Core-Plus Curriculum and while we will make recommendation, allow the individual to select when and what they will study so that by their 8th year, they can be level III certified. l have been doubled from 1,2, and 4 to 2, 4, and 8. This was done to place value on experience. It almost takes 4 years to be exposed and fully understand the cost analysis of a program. We have kept ACQ 101 and ACQ 201---colored in light blue BCF 102 and 103 will be streamlined to merge into 1XX---the same is true for BCF 205 and 211, which will be reviewed to extract core material for the BCF 2XX. Both 1XX and 2XX are our BCEFM Common courses colored in yellow/purple The courses in blue are courses that would be required at each level�those in red lettering are completely new courses. Note for Level I, we have BCF 106 (on-line and BCF 107 (resident)�this will replace the current BCF 101. For level II, the re are some modifications for 206 (Cost Risk Analysis) and 204 (intermediate cost) as well as the introduction of 4 CLMs, which are on-line short courses that could be taken in approximately 4 hours each. At level III, there is BCF 3XX, which will cover advance topics, and streamlined topics from the current BCF 301. Note also that the courses are heavily offered in the first two levels---this is done on purpose---in order to do cost estimating, you need to know a lot about a lot of different topics, we wanted to address the foundational and applicable knowledge within the first 4 years, then allow the individual to put into practice their knowledge and then come back by Level III with �total� understanding of cost and review and elaborate on cost advance topics. One final element of this program is that we want to allow each individual to select course from the �plus� part of the Cost Core-Plus Curriculum and while we will make recommendation, allow the individual to select when and what they will study so that by their 8th year, they can be level III certified.

31. Way Ahead