Surveillance Planning Team

1. Surveillance Planning Team Presentation to the Engineering Management Council February 21, 20001 Greg Robinson Revision 4

2. Purpose of team • To improve probability of mission success for NASA projects by improving Agency guidance for risk-based, functionally integrated surveillance. • Surveillance suitable to the project • Surveillance proportional to the risk • Surveillance that meets project management needs • Team Products: • Policy Direction • Procedural Guidance • Streamlined Simplified Tools to Do the Job

3. Surveillance as a method to improve likelihood of mission success • Key objectives: • Integrate surveillance approach: • Engineering, S&MA, Finance, Procurement • Improve effective utilization of surveillance resources • Enhance communication of surveillance results • Surveillance to proactively mitigate risk (part of RBAM) • Make surveillance more response to project management

4. Background (PBC Report) • July 1999, Administrator tasked the AA for Procurement to assess how the Agency could improve implementation of Performance Based Contracting (PBC) • June 2000, the PBC Team released their report. • A Significant Finding: • Many NASA Centers have not developed or implemented contract surveillance plans. • This may be due to the lack of Agency guidance for the development of meaningful surveillance planning.

5. Background (NIAT) • October 20, NASA Integrated Action Team (NIAT) Report, Enhancing Mission Success - A Framework for the Future • NIAT-13 Action Summary: • Ensure that programs and projects receive adequate surveillance based on risk knowledge and that there is ample opportunity for the communication of issues and concerns.

6. Team Sponsorship • Surveillance Planning Team: • Chartered by AA for Procurement and AA for OSMA, November 2000 • Also sponsored by the Office of the Chief Engineer • Representation: • NASA Centers: • JSC • KSC • MSFC • SSC • GSFC • NASA Headquarters: • Code Q • Code H • Code M • Code AE • Code S • Code Y

7. Team Membership • Current Members: • JSC - Jim Nise (SOMO Engineering Management) • JSC - Lee Norbraten (Management Systems Office) • KSC - Larry Tucci (S&MA) • KSC - David Culp (Procurement) • KSC- Terry Smith (Quality) • MSFC - Barry Musick (Engineering) (for Jim Kennedy) • MSFC - Charlie Chesser (S&MA) • SSC - Mike Smiles (S&MA) • GSFC - Mary Clark (EVM) • S - Nancy Porter (Program Office - Space Science) • Y - Tom Magner (Program Office - Earth Science) • M - John Castellano (Program Office - HEDS) • Q - Steve Newman (PBMA) • Q - Tom Whitmeyer (Quality) • H - Jeff Cullen (Procurement)

8. Tools To Be Used Root Cause Analysis • NIAT and PBC findings analyzed to determine underlying contributing factors. Functional Analysis • Team determines Engineering, Procurement, S&MA objectives for surveillance, and identify methods for improved implementation and integration. Benchmarking of Best Practices • Team validates approach against best-of the best in surveillance activity. Customer Coordination • EMC and PMC review.

9. Root Causes Currently Identified • Lack of meaningful guidance for conducting contract surveillance • Existing NASA Policy, Procedural Guidance, and FAR Supplementation inadequate • Need for continuous risk evaluation of contractor performance(not a one-time plan) • “Stove pipe” approach to performing surveillance • Contributing Factors • Lack of common definition • Range of contract mechanisms • Involvement of multiple functional organizations • Differences in approaches across programs and centers • Lack of agency wide training

10. Team Strategy • Define surveillance needs • Provide “risk based” options • Utilize Existing Guidance Where Possible: • NPG 7120.5 • NPG 8735.2 • NASA FAR and Supplement • Other (EVM etc) • CHALLENGES: • Large/Diverse Project Base • Large/Diverse Functional Support Organizations

11. Schedule and Products July-00 April-00 Feb-00 Jan-00 Meeting 1/9/00 Telecon 1/30/00 EMC Meeting 2/22/00 Meeting Telecon Meeting Final Report for Chief Engineer Draft NPD Initial Meeting Draft NPG For Surveillance Recommended Follow On Actions (Training, Etc.) One Week Benchmark Study Agency Objectives Publish Summary of Guidance Examples Benchmark Report Outline for NPD

12. Current Status - Next Steps • Draft NPD • Survey • NPG Initial Approach (see attached) • Benchmarking Plans • Industry • Agencies • Status meeting with PMC

13. Surveillance NPD - Key Elements • AE/Office of the Chief Engineer is responsible for the NPD, project managers are responsible for initiating and maintaining the surveillance plans. • Plans are required for contracts, grants, cooperative agreements and MOUs unless otherwise specified by the Center Director • The surveillance plan will: • Establish and maintain direct linkage with risk of the contract • Have a level of surveillance penetration that is proportional to risk areas and to the scope and risk of work to be performed. • Be a collaborative integrated effort with procurement, engineering, S&MA, and finance. • Be initiated during development of the SOW • Provide definition of government surveillance by type, responsibility, and frequency. • Specify if or how sub contract work is addressed. • Provide for the communication of surveillance plans and results in a defined manner to appropriate levels of management. • NASA engineering, procurement, safety and mission assurance, and other technical organizations shall provide surveillance support to program/project management through collaborative involvement in the development and maintenance of surveillance plans and performance of associated levels of surveillance penetration.

14. EMC BACKUP CHARTS

15. Changing Environment 80% PBC Goal “Prime” (PBC) Contractor NASA as Integrator In-House Increasing level of integration responsibility and accountability assigned to contractors

16. Cost Schedule Project Management Contract Management Technical Performance Mission Assurance Safety/ Security Project Management/Risk Management Risk Management is a Project Management Tool to Balance Requirements/Resources Functional Organizations Provide Expertise/Information to Support this process Finance Procurement Engineering S&MA

17. Fee for Results Not Best Efforts Comprehensive Surveillance Plan Contractor Quality Assurance Plan Well Defined Metrics Completion Form Contract Type (FFP, FPIF, CPIF, CPAF -- but not LOE) Performance Requirements and Standards Elements of a PBC Contract

18. • Cause and Effect Analysis Surveillance Plan • Closing a Risk • Cost-Benefit Analysis A Surveillance Plan documents • List Reduction how risks will be monitored on a • Mitigation Status Report project: the process, activities, • Multivoting milestones, and responsibilities • PERT Chart associated with surveillance. It is a subset of the • Problem-Solving Planning project plan and is written before the project begins. • Baseline Identification and Analysis • Risk Information Sheet • Spreadsheet Risk Tracking • Contractor provided metrics • Stoplight Chart • Contractor Monthly Reports • Technical Interface Meetings • Milestone Reviews • Inspections • Assessments Control • Test Reports • Bar Graph Identify • • Financial reports • EVM • Failure reports • Mitigation Status Report • Risk Information Sheet Track • Spreadsheet Risk Tracking Communicate • Stoplight Chart • Time Correlation Chart Analyze • Time Graph Plan • Baseline Planning • Planning Decision Flowchart • Planning Worksheet - Brainstorming - Cause and Effect Analysis - Cost-Benefit Analysis - Gantt Charts - Goal-Question-Measure - Interrelationship Digraph - List Reduction - Multivoting - PERT Chart - Work Breakdown Structure • Risk Information Sheet Surveillance Methods and Tools Control Identify • Project Metrics Track • Project Metrics Plan Analyze

19. Risk Drives Areas of Surveillance

20. Dynamic Tailored to selected Contractor Not incorporated into the contract Surveillance Approach Evaluated Risk Developed prior to award Insight and Oversight, commensurate with risk Based on “Risk Management” Surveillance Proportional To Risk

21. Surveillance Suitable to the Project Dynamic Low Dollar Amount Non Flight No Hazardous Operations High Dollar Amount Flight Hazardous Operations Based on “Risk Management”

22. Surveillance Dynamics (RBAM) ACQUISITION STRATEGY PROCESS REQUIREMENTS ACQUISITION STRATEGY DEVELOPMENT SOLICITATION DEVELOPMENT SOURCE SELECTION POST AWARD Contract Performance Engineering Procurement S&MA, Other Surveillance Results Preliminary Surveillance Plan Final Surveillance Plan (High/Medium Risk) based on selected contractor) RISK RANKING HIGH RISK MEDIUM RISK LOW RISK Project Manager Risk Management 22

23. Review of Process and Implementation L i k e l i h o o d 5 4 3 2 1 Did they do things right and did they do them right? Increasing technical penetration Review of Processes Did they do the right things? 1 2 3 4 5 Consequence of Failure Level of Surveillance Penetration Mapped on Risk Chart

24. No Penetration Level 0 Level 1 Level 2 Level 3 Level 4 Review of Process and Implementation Review of Processes Increasing technical penetration Level of surveillance contingent on defining an acceptable risk: • Technical risk levels • Amount of trust in performing organization’s abilities • How well processes are defined • Level at which NASA is performing Task Agreements for the program • Human rating of vehicle • Program visibility and impact of failure • Design complexity, manufacturing complexity, producibility • Value of asset Did they do the right things? Did they do the right things and did they do them right? Surveillance Continuum

25. Formulation Implementation SRR PDR CDR Production FRR Operations Acceptance/ COFR Production/ Manufacturing Package End Item Specs. System Spec. Top- Level Rqmts. Risk Plan WBS WBS WBS Trades Analysis Concepts. Detailed Drawings H/W & S/W Verification Plan Prelim. Drawings Risk Penetration Surveillance Matrix Identified Risk Identified Risk Surveillance Evolves as Risk is Better Defined

26. Technical Penetration Levels (by discipline areas) Level 0 • No Penetration • Accept performing organization’s tasks at face value (based on assessment that no penetration is required) • Contractor develops and implements verification plan Level 1 • Low Penetration • Participate in reviews and Technical Interchange Meetings and assess only the data presented • Perform periodic audits on pre-defined process(es) • Chair board or serve as board member, or RID writer, at a formal review • Participate in resolution and closure of issues • Review verification plan and its implementation • Intermediate Penetration • Includes low penetration with addition of daily or weekly involvement to identify and resolve issues • Review verification plan, its implementation, and selected verification closure data Level 2 Level 3 • In-depth Penetration • Includes intermediate penetration with addition of: • Methodical review of details • Independent models to check and compare vendor data, as required • Review verification plan, implementation, and concur in all verification closure data Level 4 • Total Penetration • Perform a complete and independent evaluation of each task • Independent review of all verification documentation (including closure data) and witness verification testing

27. Business Systems Penetration Levels Level 0 • No Penetration • Accept performance at face value (assess that no penetration is required) Level 1 • Low Penetration • Participate in or perform periodic audits on pre-defined process(es) and/or systems • Participate in resolution and closure of issues • Hold periodic status and open actions/issues reviews Level 2 • Intermediate Penetration • Includes low penetration with addition of daily or weekly involvement to identify and resolve evolving issues • Institute specific, regularly planned status and open action/issues reviews Level 3 • In-depth Penetration • Includes intermediate penetration with addition of methodical review of details • Government Senior Business Managers may participate in reviews Level 4 • Total Penetration • Perform a complete and independent evaluation of each task • Independent review of all Contractor final contract documentation • Regularly scheduled meetings of Senior Business Managers (Contractor and Government) to review current/anticipated issues and overall status

28. Technical Surveillance Plan

29. Business Surveillance Plan System/Area Surveillance Approach Estimating System Performance of audits delegated to DCMA Procurement System Performance of audits delegated to DCMA Accounting System Performance of audits delegated to DCMA Property Mgmt. System Performance of audits delegated to DCMA Major Sub-K Approval NASA review & consent on all Sub-K action over $1M Labor Relations Union contract negotiations scheduled for this summer. NASA Labor Relations Specialist involved. Periodic Floor Audits Performance of audits delegated to DCMA Corporate Financial Health Fortune 500 Company. No formal surveillance required. Socio-Economic Performance Excellent performance to date. Reviewed every 6 Months Proposal development and timely negotiation Monthly review with Senior Managers on status of all Proposal Development activity and negotiation status Performance Measurement Reviews or Audits Performance of audits delegated to DCMA Special Contract Schedules (LOE or T&M) Not applicable