DoD Title 40/CCA LSS Initiative

1. DoD Title 40/CCA LSS Initiative MOE/PIR Team Presentation 6 Nov 08 1

2. Why have MOEs and PIR been so difficult to implement? • The transition from requirements to capabilities is incomplete • “Requirements” were all about Components acquiring their systems • OT&E results were the measures of success • Capabilities are about needed effects achieved through DOTMLPF • A PIR as the assessment of DOTMLPF is the measure of success • Capabilities-based Planning (CBP) is the policy of the Department • Key is stating the capability with MOEs and measuring outcomes • Background and status of CBP • Joint Defense Capabilities Study ( Aldridge Report ) 2003 • JCIDS 3170.01C 2003 included analytic process that produced MOEs • However, MOEs not enforced by JROC • Draft 2008 JCIDS has eliminated the term MOE • Senior Warfighter Forum (SWarF) developed MOE equivalents not yet promulgated • Title 40 sec 11316, Accountability, not implemented • None of the three DoD Decision Support Systems have implemented the PIR Transparency and accountability have been difficult to achieve

3. Fielded Capabilities CBP Analysis CBA Feedback Non-materiel Solutions Acquisition PPBE Capabilities-based Planning Framework CBP from DAU Online course CLM 041

4. Role of MOEs and PIR in CBP PIR Feedback Process Fielded Capability Increments Solution Space Acquisition Problem Space CBA MOEs PPBE Should not the PIR be part of the CBP Feedback Process?

5. DOTLPF Plan and Execute Change of Processes and Culture Post Implementation Review (PIR) Assesses Outcome Of Investment By Measuring MOEs Capability Based Analysis Establishes Need and MOEs that Define Need Fulfillment CBA FRPD CD A B C IOC PIR Develop System Requirements, Acquire System and conduct OT&E on Key Performance Parameters (KPPs) PIR Information Path in CBP Feedback Process FOC M The PIR is a DOTMLPF assessment

6. MOE & PIR Summary “CCA MOE and PIR Compliance” Customers: Functional Sponsor (Complete MOE development & PIR Plan, conduct PIR, incorporate results) PM (Support assessment required by PIR Plan, incorporate results) Users (User Surveys, Satisfaction) Component and OSD (review results of the PIR) Minimum Requirements Compliance: Satisfies ability to assess degree to which need is met by DOTMLPF solution. Evidence: MOEs are linked to strategic goals Approved ICD/Business Plan containing MOEs Functional Sponsor approved PIR Plan MOE’s quantifiable and measurable/assessable Impact: MOEs/Benefits enhance AoA and raise the quality of MS A decision support PIR assesses that degree to which fielded solution meets the user needs Capabilities-based Planning does not work without MOEs and PIRs MOEs are a short hand for outcome based performance measures 6

7. MOE/PIR – Key Stakeholders Key Stakeholders in the Outcome Based Performance Measures process: Functional Sponsors Component PMO / PEO System User organizations SYSCOM / MAJCOM Capability Portfolio Manager Component / DoD CIO Identification of Functional Sponsor Upfront and Early 7

8. SIPOC Map Suppliers Inputs Process for each CBA doc until ICD approved Outputs Customers Review Phase 1 CBA doc Submit Comment KMDS Notifies POC Adjudicate comment Tel/email/ person JCIDS Components KMDS email KMDS SIPR Web site CCA compliant ICD DoD CIO/NII Components CBA Team Verify in approved ICD FCB WG Review Repeat in phase 2 Verify in CRM Input Metrics Process Metrics Output Metrics % CBA docs evaluated #C/S comments detected % CCA Compliant ICD Input process is controlled by CJCSI 3170.01 and CJCSM 3170.01 % CCA compliant ICDs Quality Speed Total IT/NSS Cost overrun X DoD CIO/NII FTEs Cost JCIDS MOE Compliance and Oversight Process Process time prescribed in CJCSM 3170.01

9. PIR Process Note: User Reps. Includes SPONSOR 9

10. MOE + PIR in the JCIDS & AMS Analyze Select Control Evaluate Materiel Solution Analysis Production & Deployment Concept Development Technology Development Engineering & Manufacturing Development & Demonstration Operations & Support MS A MS B MS C / FSDD 8. GIG 9. IA Strategy 10. Modular 11. Registration Materiel Development Decision 12. PIR Plan Capability Based Assessment 12. PIR Post-CDR Assessment CPD IOT&E ICD CDD IOC FOC 5. AoA DOT&E MDA CCA MDACCA FCB MDA CCA Update FRP Decision Review MDA JROC JROC JROC 6. EA 7. Acquisition Measures 12. Draft PIR Plan 1. Core Function Need to be Performed by Gov 2. Outcome-based Performance Measures 3. BPR analysis & Benchmarking 4. No Private or Gov better

11. MOE + PIR Data Resources & Collection Plan • MOE development rejection data collected from review of 112 CBA work products Dec 2005 through Jul 2008 • Reasons given for MOE deficiencies are based on interviews of 56 CBA team representatives • PIR information is based on AO discussions with PMO • Few programs of record had MOEs • Prior to CCA, PDPR stated but not implemented • Lack of understanding between outcome-based MOE and KPP • Confusion on roles and responsibilities • Large variance in PIR planning and execution • PIR data collection plan TBD

12. MOE Deficiencies in CBA

13. Cause & Effect Diagram: MOE + PIR

14. MOE + PIR Cause Affinity

16. Defining Potential Projects A problem statement (hypothesis): Policies surrounding MOEs are disjointed. PPBE, DAS, JCIDS should be synchronized. Describe what the potential project would fix: Would standardize and synchronize PPBE, DAS and JCIDS policies. Update the Defense Acquisition Guidebook to reflect the changes. How can the problem be measured? % of policies that are interoperable and consistent. % of new ICDs that contain outcome based performance measures. 16

17. Defining Potential Projects A problem statement (hypothesis): Training is not standardized in the area of MOEs and PIR. Describe what the potential project would fix: Once policies are updated, training would be assessed for consistency. How can the problem be measured? % of training materials that are consistent with policies. 17

18. Defining Potential Projects A problem statement (hypothesis): Currently there are insufficient business rules associated with the development of a PIR plan in the DAG or associated refined process. Describe what the potential project would fix: Develop business rules in the DAG for the PIR plan and refined process. How can the problem be measured? Incorporation of Business Rules into the DAG 18

19. Low Med High MOE + PIR Proposed LSS Project Portfolio Benefit TBD Quick Hits that are Highly Desirable Opportunities Projects in upper left are the most desirable projects. Highly Desirable Opportunities High impact Core Value Streams that require Value Stream Analysis and consist of numerous nodes of process complexity. Least Desirable Opportunities Low Med High Effort DoD CIO owns CCA Compliance & Certification Process DoD CIO does not own but participates in the process

20. BACK-UP

21. MOE/PIR – Key Success Factors • What do “successful” Outcome Based Performance Measures have in common? • PM pro-actively reaches out and ensures the Functional Sponsor is aware of the requirement, and has the latest policy and guidance • Functional Sponsor develops an early, very rough draft PIR Plan • The Functional Sponsor engages PMO, functional experts early in the draft stage • A PIR WIPT or similar stakeholder working group is involved in review/validation (not necessarily content development) • Critical content areas are addressed to include data from testing, economic calculations, qualitative assessments, combatant commander, operational, logistics, exercise data, information assurance assessments, interoperability assessments, stakeholder satisfaction and etc • MOEs are Qualitative and Quantitative and are being met “Success” is an Outcome Based Performance Measure that provides the Sponsor and program manager feedback on the program 21

22. Problem Space Solution Space DCR Draft CDD CDD CPD CBA Technology Development ICD AoA EMDD LRIP MDD MS A MS B MS C ICD PDR ICD Draft CJCSI 3170.01G JCIDS Process

23. Project Portfolio Management Lifecycle

24. AoA and Effectiveness AnalysisProcess Mission Tasks Functional Needs MOEs Materiel Approaches Initial Capabilities Document (ICD) Analysis Guidance Planning & Methodology Study Operational Effectiveness Analysis Determine Alternative Solutions Select Models And Data Perform Analysis

25. MOE/PIR Process • PIR Process essential content for compliance: • Milestone A • MOEs are established • ICD • Strategy • Milestone B add: • Draft PIR Plan • WBS, Schedule, Cost, Analysis Strategy • Milestone C, add • Final PIR Plan • Develop Metrics, Analysis, Scoring, Framework • After IOC and before FOC • Execute the PIR Plan • Data Collection, Synthesis, Trend analysis, Formatting, Presentations • Documentation, Lessons Learned, Formal Findings of PIR • Acceptance & Release • Responsibility to Implement the Recommendation and Follow-on reporting Establishing Outcome Based Performance Measures is critical to program success. 25

26. MOE/PIR Process – Key Lessons Learned • Key Lessons Learned in the Outcome Based Performance Measurement Process: • Functional Sponsors are aware of PIR Requirement Upfront • Templates eliminate guesswork and save time • A draft PIR Plan should be developed by MS B and the Final PIR Plan should be completed at MS C • MOEs should be qualitative and quantitative, and independent of any solution • Are the results of the PIR being implemented? • Realign PIR with PAE • Ensure training is aligned throughout DoDfor Performance Measures • Implement goal-oriented performance measurement Every program is special, but… 26

27. MOE/PIR – Barriers/Obstacles/Problems • Policies surrounding Outcome Based Performance Measures are disjointed. • Training is not standardized • PPBE, DAS and JCIDS Information Systems do not feed cross domain systems • There are no business rules associated with the development of a PIR plan and refined process PIR Plan -- Not perfect, but hopefully not in the critical path! 27