Disaster Preparedness Network Design Considerations

1. Disaster Preparedness Network Design Considerations Rick Spencer Qwest Staff Engineer – NCE, IEEE 2007

2. Overview • How do small/medium enterprises handle disaster recovery/business continuity (DR/BC)? • How do large enterprises and telecom carriers handle DR/BC? • Discussion: • Where do research & education (R&E) networks fall within this continuum of interests?

3. How Do Small/Medium Enterprises Handle DR/BC?

4. Qwest Business Protection Services • Common Network and IT Concerns Disaster (<1% of occurrences) Unplanned occurrences (13% of occurrences) Planned occurrences (87% of occurrences) Source: Gartner Research 2002 Survey • Disaster recovery is risk management: • Data corruption • Loss of data processing resource • Loss of access to the processing resource • Loss of people to operate the resource

5. Evolution of the DR model (early 1990s) Connected to the server through LAN Employees go to the office If disaster strikes??? Take everything and run… …to the company’s DR facility • But companies found: • Difficult to build, maintain a separate standby facility just for DR • Don’t have a facility in 2nd geographical area • Results: • Sought DR vendors…and DR vendors / services were born

6. Evolution of the DR model (mid 1990s) Lessons learned from the DR Vendors: Vendors did not have enough infrastructure to accommodate all customers at the same time DR vendors started reselling the same infrastructure to multiple customers… …and introduced a “Shared Risk Model”

7. Evolution of the DR Model (late 1990s) • Trends in the late 1990s • Users started to work from different locations – remote access became more popular • System integrators • System consolidations • ASP, ISP services more popular • Application support in often (a) outsourced (MSPs) or (b) relocated overseas • Customer implications • Role of WAN increased dramatically • Customers under-estimated the complexity and requirements of recoveries • Network management is critical (WAN, remote access, remote management…) …yet the DR models remained the same

8. How Do Large Enterprises and Telecom Carriers Handle DR/BC?

9. Disaster Recovery Fundamentals • Disaster recovery and business continuity start with the fundamentals • How is your fiber plant engineered and installed? • Conduit vs. direct buried and aerial • Where is your fiber plant? • Route risk profile • Can you find your cable in an emergency? • How well built are your facilities? • What earthquake zone are your facilities in? • Have you considered fire suppression issues? • Are the fiber entrances to your buildings diverse where available? • Do you have diversecable risers in the building where available?

10. Disaster Recovery Fundamentals • How long could you operate without commercial power? • What provisions have you made for uninterrupted power support? • DC vs. AC power • AC to DC rectifiers and battery backup • AC generators • How much is enough?

11. Fiber Plant • Fiber that is direct buried is subject to damage from concrete saws, rodents, weather, erosion, and the right of way owner planting shrubs • Direct buried cable was often buried shallow, and erosion effects (wind, rain, flooding) exposed the cable to pre-mature weathering effects • Aerial fiber is subject to micro-fractures caused by the freeze/thaw cycle, high winds, and extremely variable polarization mode dispersion (PMD)due to temperature variations

12. Fiber Plant • Qwest elected to bury high-density polyethylene (HDPE) conduit and blow our fiber through it, placing slack loops in each splice hand hole • Preference for railroad right of way locations • This route has a much lower risk profile • No one is allowed on railroad right of way without prior permission • No construction is allowed without prior notification and participation of all affected entities • Qwest’s policy is to bury conduit a minimum of 42 inches below grade • A warning tape is placed a foot above the conduit • In the event a backhoe hooked onto conduit, slack loops give the cable the ability to “give” with conduit movement, generally precluding a cable cut

13. Placing the Qwest Backbone

14. Two high-density polyethylene conduits • Buried 42"-56" below ground Installed fiber optic cable(96+ fibers) Qwest: 48 fibers Third parties: 48+ fibers Potential future fiber optic cables288+ fibers additional capacity Fiber Capacity and Expansion Capabilities

15. Entrances and Risers Kansas City, MO PoP In the best case scenario each cable would have its own manhole, entrance to the building, and its own riser shaft. In the best case scenario each cable would be fully diverse from the others, with no crossing and at least one mile separation until it approaches the building. Denver, CO Indianapolis, IN Manhole Manhole Manhole Houston, TX

16. How Do You Find Your Cable? Signal placed on the cable shield enabling a technician to locate the fiber in the field. Fiber locates constitute a major effort of the field team.

17. Facilities Issues • How old is your building? • What floor load was it designed to carry? • Was it designed with seismic activity in mind? • Which earthquake zone are you in? • It is Qwest’s policy to require all facilities to comply with zone four bracing standards • Can the building support a battery plant? • Can the building support a backup generator? • Does the building have sufficient A/C capability to disperse the predicted heat load? • Fire suppression • Water and electronics do not mix • Halon type systems require special certifications and evac plans • Check local code requirements

18. Power Considerations • What is available at your building? • Small sites, generally 240V single phase power with a 60-80 Kw backup generator • Large sites, especially cyber centers, typically include 480V three phase power, with several 2,000 Kw generators • How dependable is your local power grid? • How many outages per year can you expect and what is their worst case duration? • Qwest battery backup design standards: • Site with generator backup: 4 hours of battery capacity • Site without generator backup: 8 hours of battery capacity

19. Generators 80 Kw diesel generator set with load bank

20. Battery Banks

21. Armored Facility Grounds

22. Battery Distribution (BDFB)

23. Fiber Distribution

24. Fiber Distribution

25. Network Operations Center (NOC) • Multiple centers, geographically diversified, spread the risk • Qwest, like other carriers, has: • Two long-haul transport NOCs • Two local network NOCs • Two data system NOCs • Two government systems NOCs • This is true of backup/disaster recovery sites as well as NOCs • Geographic separation ensures that one center is available to pickup the load in the event of a regional disaster

26. Network Flexibility • Reconfigurable optical add/drop multiplexers (ROADMs) • ROADMs enable rapid, remote, circuit reconfiguration and re-routing, allowing the NOC team to take priority traffic around a disaster or nodal outage • ROADMs also enable the optical bypass that allows carriers and customers to take full advantage of the reach of the current generation of optical line systems • This translates to less equipment, lower cost and higher reliability • Next-generation tunable transceiver cards enable reduced sparing levels and faster restoration capabilities

27. Network Reliability and Survivability • Network reliability and survivability boil down to three things: • Architecture • Design around known/anticipated problem areas • Routing • Fiber • Circuits • Redundancy • Carrier grade • Critically important in power systems

28. Where Do R&E Networks Fall on This Continuum of Interests?

29. Discussion and Q&A