Systems Engineering: It’s the Law!

1. Systems Engineering: It’s the Law! An Overview of the Defense Acquisition Management System and Weapon Systems Acquisition Reform Act (WSARA) Geoff Draper Harris Government Communications Systems Division

2. Systems Engineering: It’s the Law! References: Topics: • Performance issues in Major Defense Acquisition Programs (MDAPs) • Summary of studies and reports • Overview of the Defense Acquisition Management System • DoD policies and guidance • DODI 5000.2 • Weapon Systems Acquisition Reform Act of 2009 (WSARA) • Potential Implications to the Defense Industry

3. The Problem –Improving Program Performance through SE Numerous studies and reports document program performance issues and the role of effective systems engineering • GAO-09-362T - Actions Needed to Overcome Long-standing Challenges with Weapon Systems Acquisition and Service Contract Management • “DOD’s major weapon systems programs continue to take longer to develop, cost more, and deliver fewer quantities and capabilities than originally planned.” • “costs … of major defense acquisition programs increased 26 percent and development costs increased by 40 percent from first estimates” • “programs … failed to deliver capabilities when promised—often forcing warfighters to spend additional funds on maintaining legacy systems” • “current programs experienced, on average, a 21-month delay in delivering initial capabilities to the warfighter” • NRC/USAF Study - Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Acquisition • http://www.nap.edu/catalog/12065.html • “Attention to a few critical systems engineering processes and functions particularly during preparation for Milestones A and B is essential…” • “…critically dependent on having experienced systems engineers with adequate knowledge of the domain…” • “Decisions made prior to Milestone A should be supported by a rigorous systems analysis and systems engineering process involving teams of users, acquirers, and industry representatives.”

4. Addressing SE Issues - Key Studies and Reports Details

5. Systemic Issues of Big “A” Acquisition Big “A” Acquisition* • Funding instability • Insufficient resource trade space • Budget not properly phased/magnitude to support planned development Resources(PPBE) • DefenseAcquisition • System • (DAS) Small “a” Acquisition Requirements (JCIDS) • Immature technologies • Inadequate systems engineering • Inadequate requirements flow-down/ traceability/ decomposition • Insufficient schedule trade space • Inadequate implementation of Earned Value Management System • Lack of time and assets for testing • Lack of JROC-validated requirements document for basic program (ORD, CDD, CPD) • Inadequate requirements for basic program and any increments • Critical dependence on external programs with developmental issues • Lack of inter- and intra-departmental stakeholder coordination and support Joint Capabilities Integration and Development System (CJCSI-3170) Program, Planning, Budgeting, and Execution (DoD-7000, FMRs) Defense Acquisition System (DoDI-5000) *Systemic Issues of Nunn-McCurdy “Class of 2007” Programs Synchronize JCIDS, DAS, and PPBE to deliver capabilities to Warfighters.

6. Summary of Key SE Initiatives and Reports • Other sources: • Program Support Reviews (PSRs) • Nunn-McCurdy breaches • Congressional oversight • Defense Science Board • Other studies, reviews, reports Common program issues: -Poor program planning -Unrealistic estimates -Unstable requirements -Immature technology -Not following SE processes -Insufficient SE expertise -Ineffective reviews -Poor system reliability 2010 2003 2006 2008 2009 • NRC/USAF Study: Pre-MS A and Early SE [1/08] • DODI 5000.2 updates [12/08] • Systemic Root Cause Analysis (SRCA)[draft] • NDIA Top 5 • SE Issues [1/03] • DODI 5000.2 [5/03] • PL 111-23 (WSARA) [5/09] • DTM-09-027 Implementation of WSARA [12/09] • Defense Acq Guide (DAG);Chap 4: SE [7/06] • OSD policy, guidance, reviews: SEP; IMS; training; risk mgmt; PSRs • NDIA Top SE Issues [7/06] • NDIA Top SW Issues [9/06] • QDR [2/10] • House Armed Services Committee report [3/10] • OSD WSARA report to Congress [3/10] NDIA Reports DoD policy and actions Other (Congress, studies, reviews) There is no shortage of sources confirming we have issues – How do government and industry collaborate to act upon solutions?

7. Secretary of Defense Direction Chief among institutional challenges facing the Department is acquisition.”

8. Weapons Systems Acquisition Reform Act “The key to successful acquisition programs is getting things right from the start with sound systems engineering, cost estimating, and developmental testing early in the program cycle. The bill that we are introducing today will require the Department of Defense to take the steps needed to put major defense acquisition programs on a sound footing from the outset. If these changes are successfully implemented, they should help our acquisition programs avoid future cost overruns, schedule delays, and performance problems.” –Senator Carl Levin, Chairman, Senate Armed Services Committee “The Weapon System Acquisition Reform Act of 2009 is an important step in efforts to reform the defense acquisition process. This legislation is needed to focus acquisition and procurement on emphasizing systems engineering; more effective upfront planning and management of technology risk; and growing the acquisition workforce to meet program objectives.” –Senator John McCain, Ranking Member, Senate Armed Services Committee

9. P. L. 111-23: Weapon Systems Acquisition Reform Act (WSARA) – May 2009 • Key Elements of Legislation: • Organizational • Establishes Directorates for SE and DT&E as principal advisors • Joint tracking of component and MDAP progress against plans and measurable criteria with annual reporting to Congress • Independent cost estimation and cost analysis (Director, CAPE) • Provide adequate trained staff for SE and development planning • Conduct MDAP performance assessments and root cause analysis • Role of SE across program lifecycle • Developmental planning, lifecycle management, sustainability • Ensure reliability, availability, maintainability (RAM) • Mandates • Measurable performance criteria in SE/DT&E plans • Competitive prototypes for MDAPs; prime make/buy analyses • System PDR before MS B, with formal MDA assessment • Assessment of technical maturity and integration risk of critical technologies during Technology Development (TD) • Technical and cost oversight • Independent estimates, Problem Assessment Root Cause Analysis • Technical analysis of cost/schedule breaches; presumed termination http://thomas.loc.gov/cgi-bin/query/z?c111:S.454.enr

11. Key Acquisition Business Process Changes CDD P-PDRA PDR X • Increased Emphasis on Milestone A • Mandatory for MDAPs with Technology Development Programs • Likely for Most Programs When PDR is Conducted after Milestone B an MDA Post-PDR Assessment is Required Preliminary Design Review (PDR) before Milestone B to Enhance Understanding of Derived Requirements and Improve Cost Estimation Post-Critical Design Review Assessment –A Mandatory Decision Point to Review Progress MS A MS B MS C FRP DR X ICD MDD Materiel Solution Analysis PCDRA Engineering & Manufacturing Development Technology Development PDR Production & Deployment Operations & Support CPD Competitive Prototyping Re-structured “EMD” Phase Effective Contracting via Pre-Award Peer Reviews Materiel Development Decision – Mandatory Process Entry Point • Enhanced Emphasis on: • Technology Maturity • Systems Engineering • Integrated Testing and Test Planning • Manufacturing and Producibility • Logistics and Sustainment Planning Competitive Prototyping Configuration Steering BoardsEstablished to Stabilize Requirements

14. Technology and Manufacturing Readiness A B C IOC FOC Materiel Solution Analysis TECHNOLOGY DEVELOPMENT Engineering & Manufacturing Development PRODUCTION & DEPLOYMENT OPERATIONS & SUPPORT FRP Decision Review Materiel Development Decision Post CDR Assessment Technology Readiness Levels Defense Acquisition Guidebook para. 10.5.2 TRLs 1-3 TRL 4 TRL 5 TRL 6 TRL 7 TRL 8 TRL 9 Analytical/ Experimental Critical Function/ Characteristic Proof of Concept Component And/or Breadboard Validation In a Laboratory Environment Component And/or Breadboard Validation In a Relevant Environment System/ Subsystem Model or Prototype Demonstrated In a Relevant Environment System Prototype Demonstrated In an Operational Environment Actual System Completed Qualified Through Test and Demonstration Actual System “Mission Proven” Through Successful Operations MRL 4 MRL 5 MRL 6 MRL 7 MRL 8 MRL 9 MRL 10 MRLs 1-3 Manufacturing Readiness Levels Draft MRA Deskbook May 2008 Manufacturing Feasibility Assessed. Concepts defined/ developed Capability to produce Technology In Lab Environment. Manufacturing Risks Identified Capability to Produce Prototype Components Capability to Produce System/ Subsystem Prototypes Capability to Produce Systems, Subsystems Or Components in a Production Representative Environment Pilot Line Capability Demonstrated. Ready for LRIP Low Rate Production Demonstrated. Capability In Place for FRP Full Rate Production Demonstrated. Lean Production Practices In Place Manufacturing Cost Drivers Identified Cost Model Constructed Detailed Cost Analysis Complete Cost Model Updated To System Level Unit Cost Reduction Efforts Underway Engineering Cost Model Validated LRIP Cost Goals Met Learning Curve Validated FRP Unit Cost Goals Met Section 2366b of Title 10, United States Code, requires certification that: the technology in the program has been demonstrated in a relevant environment to enter Milestone B. [TRL 6]

15. DoD Strategic Management Plan (7/09)Key acquisition-related excerpts http://dcmo.defense.gov/documents/2009SMP.pdf

16. Areas of emphasis: Defense Strategy Defense objectives, emerging threats Rebalancing the Force Counterinsurgency, counterterrorism, WMD, cyber U.S. force structure: sizing, shaping, evolution Defense workforce Supporting troops, deployment Recruiting, retention, development Strengthening relationships U.S. defense posture, interagency, abroad Reforming How We Do Business Rapid acquisition, security, how we buy, exports Strengthening the industrial base Strategic approach to climate and energy Defense Risk Management Framework Operational, force management, institutional, future challenges, strategic, military, political What’s Next? http://www.defense.gov/QDR/ A U.S. force “prepared to conduct a wide variety of missions under a range of different circumstances.”

17. Summary DODI 5000.2 and WSARA are changing the game Early life cycle planning (SEP) Adherence to effective SE practices Independent cost estimates Competitive prototyping Managed technology risks (TRL) Increased emphasis on reliability and supportability (RAM) Congressional reporting and oversight More fixed priced contracts and evolutionary acquisition likely Revitalization of DoD SE and acquisition workforce Questions? Geoff Draper (gdraper@harris.com, 321-727-5617)

18. Backup

20. Addressing SE Issues – Key NDIA Task Groups and Milestones • Government Studies/Reports (GAO, NRC, DCMA, etc.) • Reviews (QDR, etc.) • Program Support Reviews (PSRs) • Congressional oversight (Nunn-McCurdy, McCain-Levin, …) • Working Groups • Conferences • Forums 2003 2006 2007 2008 2009 • NDIA Top 5 • SE Issues • NDIA Top 5 SE Issues • NDIA Top SW Issues • Defense Software Strategy Summit • SE Effectiveness Survey • Systemic Root Cause Analysis (SRCA)[draft] • Industrial Committee on Program Mgmt (ICPM) • SE of Tactical Air Launched Systems (Gulf Coast Chapter) (Air Armament Center) • Top SE/SW Issues Update(planned) • Industrial Committee on SE (ICSE) (planned) Common program issues: -Poor program planning -Unrealistic estimates -Unstable requirements -Immature technology -Not following SE processes -Insufficient SE expertise -Ineffective reviews -Poor system reliability There is no shortage of sources confirming we have issues – How do government and industry collaborate to act upon solutions?

21. NRC Study –Pre-MS A and Early-Phase SE • Reference: Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Acquisition.http://www.nap.edu/catalog/12065.html

22. NDIA Top 5 SE IssuesNDIA SE Division (July 2006) Provides an update and status from a previous task group report conducted in 2003. Reference: NDIA Top 5 Systems Engineering Issues Report. July 2006. NDIA Systems Engineering Division. [report, briefing] http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Pages/Past_Projects.aspx

23. NDIA Top Software IssuesNDIA SE Division (Sept 2006) Reference: NDIA Top Software Engineering Issues Report. Sep 2006. NDIA Systems Engineering Division. [report, briefing] http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Pages/Past_Projects.aspx

24. Systemic Root Cause Analysis (SRCA)- Conducted 2007-2008, publication pending • SRCA workshops and task group • Based on PSR findings (44 programs) • Tagged to core/systemic root causes • 95 preliminary recommendations • 48 systemic issues • 3 Recommendation Areas * Reference SRCA briefing for additional details and recommended government actions.

25. SE of Tactical Air Launched WeaponsNDIA Gulf Coast Chapter (2008) Reference: “Systems Engineering of Tactical Air-Launched Weapons: An Industry Examination.” Air Armament Symposium, 2008. [briefing] http://www.ndiagulfcoast.com/events/archive/34th_Symposium/34_Day1/11_SysEngNDIASymposiumBriefOct208.pdf

26. A Survey of Systems Engineering Effectiveness (2007) • Joint NDIA/SEI survey of 46 programs correlating the effectiveness of SE processes with program performance. • SE processes most strongly correlated with better program performance: • Architecture • Trade Studies • Technical Solution • IPT Capability • Requirements Development and Management • Report: http://www.sei.cmu.edu/publications/ • documents/08.reports/08sr034.html Projects with better Systems Engineering Capabilities deliver better Project Performance (cost, schedule, functionality)

27. Defense Acquisition System Weighted Expenditures DoDI 5000.02 Perspective ProgramInitiation A C B IOC FOC MaterielSolutionAnalysis Engineering and Manufacturing Development Technology Development Production & Deployment Operations & Support FRP Decision Review Materiel Development Decision Post-CDR Assessment LRIP/IOT&E Pre-Systems Acquisition Systems Acquisition Sustainment Warfighter and Sustainment Organization Perspective (Program Initiation) A B C Materiel Solution Analysis Technology Development Engineering and Manufacturing Development Production & Deployment Operations & Support FR5P Decision Review Materiel Development Decision LRIP/IOT&E Post-CDR Assessment Pre-Systems Acquisition Systems Acquisition Sustainment 65-80% 20-35% 30+ YEARS Nominal Life Cycle Cost Distribution