1 / 42

Seeking Your Contingency Plan: Are You HOT , COLD , or WARM?

Seeking Your Contingency Plan: Are You HOT , COLD , or WARM?. NCHICA AMC Security and Privacy Conference September 26-28, 2005 Panel Members: Bill Rider, Panel Leader, Mgr. Information Security and Disaster Recovery, Johns Hopkins Health System and University

britanni-fields
Télécharger la présentation

Seeking Your Contingency Plan: Are You HOT , COLD , or WARM?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Seeking Your Contingency Plan: Are YouHOT, COLD, or WARM? NCHICA AMC Security and Privacy Conference September 26-28, 2005 Panel Members: Bill Rider, Panel Leader, Mgr. Information Security and Disaster Recovery, Johns Hopkins Health System and University Anne Marie Turner, Information Systems Risk Mgr., University of Rochester Medical Center

  2. Agenda • Overview of two Institutions • Demographics • IT Divisions • DR Theories – Baseline Strategies for Determining HOT, COLD, or WARM • Institution DR Practices – Comparisons • Mainframe, Open Systems • RTO and RPO • Underlying factors: internal vs. external • Testing • Maintenance • Are You HOT, COLD, or WARM: • RPO and RTO Considerations • Results • Questions to Consider • Overview: Healthcare BCP/DRP Benchmarks • Questions & Answers

  3. Johns Hopkins Health System & University Acute Care Beds Licensed 1,467 Opened 1,327 % Of Occupancy 77.0 Discharges 77,962 Days 381,601 Outpatient Encounters 1,929,660 Emergency Visits 208,682 Operating Room Cases 68,070 Inpatient 27,968 Outpatient 40,102 (Johns Hopkins Hospital, Bayview Medical Center, Howard County General Hospital, Johns Hopkins Clinical Physicians, JHU School of Medicine)

  4. Johns Hopkins Hospital & University I.T. Environment 1 IBM Mainframe Processor 406 MIPS (DR is 206) 8 Meg Memory (DR is 4) 1.3 Terabytes Storage (DR is 1.3 Tb) Multiple DEC and AIX Midrange Platforms Over 450 Distributed Servers Multiple I.T. locations within 15 mile radius Organization segregates: I.T. Operations Network Services Enterprise Services Applications Support

  5. Number of Beds: Strong 750 beds, Highland 275=1025 Number of In-Patient Discharges per year: 55K (SMH and HH) Number of Out-Patient Discharges per year: Outpatient Visits (not discharges): SMH OP: 360K UR Medical Faculty Group: 660K SMH Emergency Dept 90K SMH Lab Specimens 500K (these are not actual patient visits but do result in registrations) Eastman Dental Center –                          50K HH Emergency Dept -                        25K HH OP -                        75K Total 1.7M Strong Memorial Hospital, Highland Hospital, School of Medicine & Dentistry, School of Nursing, Research, Mt. Hope Family Center, UR Medical Faculty Group, University Health Service, Primary Care Network, Eastman Dental Center, Long Term Care, Visiting Nurse Service

  6. University of Rochester Medical Center Information Systems Division • IBM Mainframe Processor • Multiple AIX Midrange Platforms • Over 280 Open System Medical Center Servers in Data Center • Information Systems Division: • Enterprise Operations • Network Services • Enterprise LAN • Security • Help Desk Support / Desktop Support • Systems Interface • Research Support • Shared Services Organization • Medical Informatics • Clinical Systems Applications • Project Management

  7. Are You HOT, COLD, or WARM? RTO & RPO RECOVERY TIME OBJECTIVE: (RTO) The period of time in which systems, applications, or I.S. functions must be recovered after an outage. RTO's are often used as the basis for the development of recovery strategies, and as a determinant as to whether or not to implement the recovery strategies during a disaster situation. RECOVERY POINT OBJECTIVE: (RPO) The point in time to which systems and data must be restored after an outage. RPO's are often used as the basis for the development of backup strategies, and as a determinant of the amount of data that may need to be recreated after the systems or functions have been recovered. I.T. RTO and RPO Need To Be Negotiated With The Business Units….In Order To Provide A Level Set For User Expectations Regarding The Current Version Of The Data, The Timely Restoration Of The Systems, And For The Business Units To Accurately Develop Their Business Contingency Plans

  8. Comparison of $$ Impact To Cost To Recover * $$ Cost of Impact 900,000 800,000 700,000 600,000 500,000 Optimum Recovery Point (i.e., Cost vs. Risk) 400,000 300,000 200,000 100,000 50,000 Cost of Recovery * 0 30 min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - 0 - 30 mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs * Numbers Above Are For Demonstration Purpose Only Accurate Impact numbers can be determined from the Business Impact Analysis Are You HOT, COLD, or WARM? The Bottom Line for Recovery

  9. Dispersed Processing – Open & Central Systems Are You HOT, COLD, or WARM? Print Server Application Server EPR Database Mainframe Pharmacy Database CITRIX Server Database Server PID Mainframe Clinweb Clindata Outpatient Reg Sybase Server Inpatient Reg Mainframe Clinweb Outpatient Reg Clindata Sybase Server Outpatient Reg

  10. Recovery Strategy Needs to Recognize Diversity Are You HOT, COLD, or WARM? Print Server Application Server Pharmacy Database EPR Database Mainframe CITRIX Server Database Server PID Mainframe Clinweb Clindata Outpatient Reg Sybase Server Internal Solution External Solution Inpatient Reg Mainframe Clinweb Outpatient Reg Clindata Sybase Server Outpatient Reg Open Systems DR Model Centralized DR Model

  11. Print Server Application Server Pharmacy Database EPR Database Mainframe CITRIX Server Database Server PID Mainframe Clinweb Outpatient Reg Clindata Sybase Server Inpatient Reg Mainframe Clinweb Outpatient Reg Clindata Sybase Server Outpatient Reg Are You HOT, COLD, or WARM? Different Environments, Tactics, & Controls • Centralized DR Model • Traditional Offsite Storage • Hotsite Location Approx 100 Miles From Primary Site • T1 Connection Between Hotsite and Local Internal Solution • Open Systems DR Model • Data Replication To Local Storage • Failover and/or Quick Recovery • Local Connections For High Volume • Local AND Remote Recovery

  12. Mainframe and Open Systems URMC • Multiple Platforms • Mainframe/Midrange/Servers • Centralized Locations • Primary Data Center/Backup Data Center/Hospitals • Strong Health Geography • SMH, HH, and Data Centers within 5 miles • Outlying Locations • Urban, Suburban, and Rural Hopkins • Multiple Platforms • Mainframe/Midrange/Servers • Dispersed Locations • Hospital/University/Clinical Practices/Satellite JHU Campuses • Campus Geography • JHH and JHU within 10 miles • Outlying Locations • Urban and Rural

  13. RPO, RTO, Outsourced vs. In-House URMC • Criticality / BIA Driven: • Patient Safety • Patient Quality of Care • Workforce Productivity • Financial • Legal/Regulatory • Reputation • Service Level Agreements • Educational • Research Programs • Mission Critical: • Dedicated DR systems & data in backup data center • Offsite media • Business Critical: • Contract for shipment • Offsite Media Hopkins • Platform Size Driven • Mainframe and Midranges at Hotsite • Servers In-House

  14. Underlying Factors: Internal vs. Outsourcing Hopkins • RTO • RPO • Cost Of Recovery • Impact Of Recovery URMC • Mission Critical • RTO • 2 to 24 hours • RPO • 0 to minimal data loss • BIA • Downtime Impact vs. Time to Recover Costs • Dedicated DR Hardware • DR Contract for Shipment

  15. Testing: HOT, WARM, COLD, Internal vs. Outsourcing Hopkins • Combination • Relocation to hotsite – 2 per year • Servers tested throughout the year – validated during hotsite tests • Customers test locally • High volume applications (email, images, etc.) test locally URMC • Mission Critical Systems • 2 per year • Business Critical Systems • 1 per year • Systems tested throughout the year: • Hardware and Software Upgrades, project implementations, etc. • Department Downtime Procedures – Annual Testing

  16. Maintenance: HOT, WARM, COLD, Internal vs. Outsourcing Hopkins • All maintenance issues validated during hotsite tests • In-House & External Contracts • Documentation • Contact Lists • Skill Sets • Notification, Escalation, Declaration Procedures URMC • Disaster Recovery System and Program Maintenance validated through: • In-House DR system testing, IT and End-User testing • Technical Documentation Audits • Skill Set Evaluations • Internal and External Auditors • EOC and ISD Command Center Team Exercises • Change Management • Project Management • Incident Debriefing Sessions

  17. Are You HOT, COLD, or WARM? RTO & RPO Considerations Negotiate The Service Level Agreement Between I.T. And Business Operations: • Use Both The I.T. And Business RTO & RPO As The Basis • Disaster Recovery Plan Test Results Quantify Timelines • Business Contingency Plan Exercises Qualify Impact • I.T. Capabilities Improve Timelines – But At A Cost • Business Contingencies Reduce Impact - But Require I.T. Capabilities • Criticality Rankings • Systems Recovery Sequencing • Business Process Prioritization • I.T. and Business Process Timelines • Negotiated RTO and RPO

  18. Are You HOT, COLD, or WARM? Results I.T. • Better Understands the Customers’ Issues and Requirements • Obtains A Clearly Documented Set of Customer Expectations for DRP’s • Clarify and Justify Budget Forecasts • Establishes Specific Test Objectives • Ensure Active Customer Involvement in Testing & Recovery Processes Business Units • Better Understands the Role Of I.T. in the Contingency Process • Obtains a Set of Parameters from which to Develop Their BCP’s • Workaround Procedures During Downtime • Procedures For Capturing Lost Transactions From Downtime and During Recovery • Restoration of Normal Environments Everyone works towards a common interest, that of ensuring that the business processes of the organization, its mission, goals, and objectives…..and possibly the community at large…are protected

  19. Are You HOT, COLD, or WARM? Questions To Consider • Was the original disaster recovery initiative driven by I.T., business units, or Sr Management ? • What are Sr. Management’s expectations with respect to continuity of service ? • Has a business impact analysis been done on some or all of the business units ? • Quantified Impact • Quantified Cost of DRP vs. Impact of Risk • Acceptable Downtime Criteria (services, workstations, etc.) • What discussions have taken place between I.T. and critical business units ? • State of DRP • State of BCP • Quantified RTOs and RPOs • Systems Development Life Cycles

  20. Are You HOT, COLD, or WARM? Questions To Consider • What are the business units’ expectation with respect to current I.T. RTOs and RTOs ? • Are they driven by I.T. technologies or business requirements ? • Are there current SLAs ? • Service Center • Problem/Change Control • Network Outage Response Time Are regulatory compliance, industry certification, or audit issues creating more compelling reasons for addressing DRP and BCP ?

  21. 40% of Members responded to Benchmark Survey 58% of Responders have Steering Committees Benchmark Survey Results 83% Feel they were prepared for the HIPAA deadline 75% Have DR Plans for Critical Systems 67% Use Planning Software 58% Have performed a BIA in last 12 months

  22. Benchmark Survey Results Disaster Recovery Plan Testing 25% Test Every Six Months 25 % Test Annually 50% Currently Developing A Testing Strategy

  23. Benchmark Survey Results Percentage of IT Budget Spent for Disaster Recovery <1% 1% 3% 5%

  24. Benchmark Survey Results Organizations with Recovery Solutions 35% 30% 25% Own 20% Co-Lo 15% Contract No Response 10% 33% 33% 26% 5% 8% 0% Percentage with Recovery Solutions

  25. Benchmark Survey Results Minimum Recovery Timeframes for Critical Systems 35% 30% 33% 25% 25% 20% 15% 17% 17% 10% 8% 5% 0% Under 12 Hrs. 12-24 Hrs. 24 Hrs. 48-72 Hrs. Not specified

  26. Business Continuity Planning Workgroup for Healthcare Organizations

  27. BCPWHO Information • 25+ Academic Medical Centers and Healthcare Organizations • Charter and Bylaws in Progress • Website – Coming • Dedicated BCPWHO Chat Room for Questions, Issues, Discussion – Coming • Opportunities: National Meetings and Regional Workshops • DRP/BCP Logistical and Vendor Sponsorships – Establishing and Growing • Vendor Resources – Communication & Awareness

  28. Interest Level Survey Results • 60 Surveys Sent Out • 22 Surveys Received Back • 21 Clinical • 13 Academic • 10 Research • 3 Disaster Recovery Planning Only • 16 Combined Disaster Recovery and Business Continuity Planning • 3 Were Other (Insurance)

  29. Interest Level Survey Results • Majority Of Responders Agree: • Formal Membership • Membership Dues • Charter With Bylaws • Formal Board With Rotating Members • Some type of regional users groups • Tie annual meeting to conference venue • DRJ CPM CI Strohl • Virtual Workshops (Bi-monthly or Quarterly) • Internet Chat Rooms With Participation Limited To Members Only

  30. Interest Level Survey Results And Who Are We All Reporting To ? I.S. Security Director Director of Technical Services VP Information Services Director, Facilities & Safety Programs Chief Security Officer Assoc. Vice Chancellor Health Affairs, Director Of Informatics Center V.P Information Services CIO Assoc. Vice Chancellor Health Affairs, Director Of Informatics Center Sr. Vice President, Quality Care and Chief Medical Officer Director of Engineering Services, Enterprise Technology Services Executive Associate Dean for Faculty Affairs Chief Technology Officer Senior Director SR VP Legal Affairs and HR

  31. About BCPWHO What Is BCP Toolkits Industry Resources Members Only Contact Us > > > > > > > > > > > > Our Sponsors The Continuity Management Program C Levels’ Corner Regulatory & Compliance Corner > More Info Emergency Response Planning Disaster Recovery Planning Business Contingency Planning Crisis Management Planning > More Info > More Info > More Info > More Info Upcoming Events News Industry Certifications & Associations Business Continuity Disaster Recovery Emergency Management Data & Network Security Users Groups Conferences Training Sessions Local National International

  32. Academic Medical Center / Hospital / Healthcare Organization Name: Contact Name and Title: Contact Phone Number: Contact E-Mail Address: Business Continuity Planning Workgroup for Healthcare Organizations Spring 2005 Disaster Recovery Benchmark Survey To BCPWHO Members: Please complete the healthcare disaster recovery benchmark survey below and return the survey back to Kathy Lee Patterson, pattersonkl@email.chop.edu, of the BCPWHO Planning Committee by April 8, 2005. The information provided will be held in strictest confidence, with published results sent only to BCPWHO members. Healthcare establishment’s names will not be published, only the statistical results. General results of the survey will be presented at the May Continuity Insights Conference in New Orleans during the BCPWHO session (C9, Healthcare/AMC DRP-BCP Consortium: A Whole Different Challenge). Thank you for your participation. (While this survey is generally I.T. DRP focused, future surveys will be developed to address other areas of Business Continuity Planning) Hyperlink to benchmark survey

  33. Seeking Your Contingency Plan – Are You:HOT, COLD, or WARM? Questions?

  34. Engagement Process • Facilitators: • Stimulate audience discussion with: • requests for questions and comments , • Pre-designed questions and “instant polls” that are designed to assess how the audience of AMC peers sees the topic and to start further questions and comments from the audience. • Collect the results for reporting in the “track reporting” part of each plenary session and a planned GASP (Guidelines for AMCs on Security and Privacy) update. • Audience (and panelists): Respond to the questions, comments, provide your own.

  35. Instant Poll Rules • Facilitators role: • Require audience members and panelists to shut their eyes (to promote more honest voting) • Ask for a show of hands for each item to be voted on. • Audience role: • Vote as you see fit. • Voting is anonymous. • Follow-up questions may ask voters to describe why they voted as they did, if they are comfortable doing so. • Anonymity: • For some issues, you may wish to keep your vote private; the “eyes-shut” voting rule is the main rule that assures this. • Also, the facilitators will take only the notes that you see on the screen and will not identify you by name or institution unless you explicitly say that you are willing to be so identified.

  36. Instant Poll Rules • Facilitators role: • Ask audience members and panelist to shut their eyes (to promote more honest voting) • Ask for a show of hands for each item to be voted on. • Audience role: • Vote as you see fit. • Voting is anonymous. • Follow-up questions may ask voters to describe why they voted as they did, if they are comfortable doing so.

  37. Conference Benchmarks -Disaster Recovery Planning • My AMC tests its disaster recovery plan every six months _____ • My AMC tests its disaster recovery plan every year _____ • My AMC has not tested its disaster recovery plan ____ • My AMC is still developing its disaster recovery plan ____ • Active test plans ___

  38. Conference Benchmarks - Minimum Recovery Timeframes for Critical Systems Based on Some level of BIA • Under 12 hours ____ • 12-24 hours ____ • 24 hours ___ • 48 - 72 hours ____ • Not specified _____

  39. Contingency Planning - Discussion • Should there be uniform standards? • Should the government help pay? • Is contingency planning a public health issue? • What can AMCs be doing better?

  40. Contingency Planning • What aspects of contingency planning have not been adequately addressed today? • What have you heard today that you want to pursue further? • Any surprises in what you heard today?

  41. What follow-up activities would be helpful to AMCs in dealing with this topic?

  42. Engagement Quality Instant Poll • This session did a good job of engaging the panelists and the audience on the topic. 1 - Strongly Disagree ___ 2 - Disagree ___ 3 - Neither agree not disagree ___ 4 – Agree ____ 5 - Strongly agree ____

More Related