Breakout Session 802

2. Procurement in the Networked World: A Case Study in Disaster Preparedness Breakout Session 802 Ms. Ruby Harvey, Chief, Procurement Systems Division, United States Department of Agriculture Mr. Frank Palmer, Senior Associate, Booz Allen Hamilton Date: April 10, 2006 Time: 1:30 – 2:30 PM NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

3. Overview • USDA’s Integrated Acquisition System Overview (15 minutes) • Scope, Process, Technology • Disaster Recovery Case Study (30 minutes) • Preparation and Planning • Scope and Frequency • Outcomes • Hurricane Katrina • Disaster Recovery Activation • Technical and Data Challenges • System Restoration and Data Re-synchronization • Lessons Learned • Question & Answer (15 minutes) NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

4. USDA’s Integrated Acquisition System (IAS) Overview • Web-based enterprise-wide COTS (commercial-off-the-shelf) • e-Procurement and contract management solution • Integrated with financial system • Replaces 5 individual agency contract writing and purchase order systems across 10 agencies with administrative procurement authority • Will reach approximately 20,000 Desktops • 2005 Fed100 Award Recipient NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

5. Functional scope of IAS IAS Functional Vision • Core Functionality complete with USDA financial interface • Moving towards Extended Functionality e-Catalog Asset Management Interface e-Invoicing Requisition Processing Contract Management Data Warehouse e-RFx Auctions NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

6. Payment Backfeed Order / Contract Requisition Receipt Invoice IAS supports a standard, Enterprise-wide Procurement process • IAS supports the FAR and USDA-specific (AGAR) buying process • Allows for agency-specific settings within these broad parameters • IAS enforces business and policy rules • Requires collaboration among acquisition, program, technical and other administrative teams within and external to agencies • Provides opportunities to streamline operations within the organization NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

7. The IAS is comprised of a network-of-networks on USDA’s backbone NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

8. Overview • USDA’s Integrated Acquisition System Overview (15 minutes) • Scope, Process, Technology • Disaster Recovery Case Study (30 minutes) • Preparation and Planning • Scope and Frequency • Outcomes • Hurricane Katrina • Disaster Recovery Activation • Technical and Data Challenges • System Restoration and Data Re-synchronization • Lessons Learned • Question & Answer (15 minutes) NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

9. Disaster Recovery Case Study Lesson Learned #1: Disaster Recover drills are essential to understanding Roles and Responsibilities and establishing command and control • Preparation Planning • USDA’s Procurement Systems Division (PSD) has been participating in semi-annual disaster recovery exercises since 2003 to assess organizational and system readiness in the event of a man-made or natural disaster • The drills focused on the ability to “standup” mission critical applications, establishing system connectivity, verbal communications and roles and responsibilities in preparation of a disaster • While the drills did not address the complexity associated with the real-time loss of a major component of the procurement system, it established a solid foundation for which critical operations were to be restored and in what order NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

10. Disaster Recovery Case Study (cont) • Preparation Planning Outcomes • PSD refined the DR Plans along with OCFO each year as we improved our disaster recovery capabilities and accounted for changes in configurations and complexity • The results of the planning allowed USDA to make refinements and updates to DR procedures in an iterative fashion, establishing deeper understanding of technical challenges and areas in need of focus • The number of stakeholders involved in DR planning and execution can differed from what is necessary during the recovery period • Run through various disaster scenarios other than the ones that are planned to determine if additional stakeholders emerge • Determine if any competing priorities arise that may also uncover additional stakeholders NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

11. Overview • USDA’s Integrated Acquisition System Overview (15 minutes) • Scope, Process, Technology • Disaster Recovery Case Study (30 minutes) • Preparation and Planning • Scope and Frequency • Outcomes • Hurricane Katrina • Disaster Recovery Activation • Technical and Data Challenges • System Restoration and Data Re-synchronization • Lessons Learned • Question & Answer (15 minutes) NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

12. Hurricane Katrina – August 2005 • Disaster Recovery Activation • When USDA NFC declared a disaster, financial systems operations ceased at NFC and DR activities were activated to recover the financial systems operations at the alternate site • Because IAS was designed in a decentralized architecture (separate locations for the procurement system and financial systems), the procurement system remained up and running (queuing new transactions) while the financial system was being restored NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

13. Lesson Learned #2: A distributed technical architecture minimized impact to the procurement community when the financial system was shut down Hurricane Katrina (cont) USDA IAS Architecture • Technical Challenges • While the DR planning was extremely beneficial to smooth communications, roles and responsibilities and coordination, technical challenges were soon realized once the applications were restored at the alternate site • The financial system had to be taken off-line during normal operating hours in order to allow processing of other mission critical applications Backup Site for Financial System - Primary Site - Alternate Site - Loss of critical function Primary Site for Financial System NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

14. Lesson Learned #3: Differences in Hardware and Infrastructure at backup sites can impact recovery times and normal operations Hurricane Katrina (cont) • System Restoration and Data Resynchronization • While the DR planning was beneficial from a coordination perspective, it did not (could not) prepare the technical team for the technical challenges they would have for restoring full system availability for several weeks • If hardware and Infrastructure redundancy between primary and backup sites are not identical, it could result in sporadic times of unavailability, limited queue capacity and throughput challenges • Depending on the configuration of your solution, your data will be backed up at established intervals • Procurement Systems Division had to replay procurement related transactions in order to resynchronize the procurement and financial data from the last time data was backed up to the time the systems were taken off-line which was a time consuming event • Robotics are an integral part of normal operations and maintenance – Staff at alternate site must deal with competing priorities and fatigue if robotics are not in place NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

15. Lesson Learned #4: No two disaster are the same. Think through possible scenarios and prepare for the most probable ones Lessons Learned • DR planning provided much need understanding of Roles & Responsibilities when the disaster struck • Having a distributed web-based enterprise solution allowed the procurement community to process procurements (queuing) during the recovery period at year-end • Think ahead as to how you will restore system data under the worst circumstances and the time it will take • No two disasters are alike, be prepared to address situations that are not planned for NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

16. Lessons Learned (cont) • Having like hardware and infrastructure at the primary and backup sites • Maintaining current Standard Operating Procedures (SOP’s) and CM records at both sites will smooth the transition at the backup site • The best recovery plans will suffer when recovery team members are busy responding to human factor impacts and are unavailable to do other work • A standing 24x7 conference call line is very useful • Teamwork is key, tensions will run high…get plenty of rest. • End to end recovery drills are important but it is equally important to run contingency drills that test one or two major objectives • Have a plan to recover your development and quality assurance environments NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

17. Lessons Learned (cont) • Keep your Business Impact Analysis (BIA) up to date, it will drive your disaster recovery plan • There is always a workaround • Good executives always ask for identified risks, impacts and options before committing to a corrective action plan • Keep a running chronology of events during the recovery • If your recovery environments are not the same as your production environment, be prepared for performance issues • Load test your recovery environment NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management

18. Questions? NCMA World Congress 2006 : Achieving High Performance in Global Business: Leadership, Outsourcing, & Risk Management