1 / 28

Masking Failures from Application Performance in Data Center Networks with Shareable Backup

Masking Failures from Application Performance in Data Center Networks with Shareable Backup. Dingming Wu + , Yiting Xia +* , Xiaoye Steven Sun + , Xin Sunny Huang + , Simbarashe Dzinamarira + , T. S. Eugene Ng + + Rice University, * Facebook, Inc. Data Center Network Should be Reliable.

donaldv
Télécharger la présentation

Masking Failures from Application Performance in Data Center Networks with Shareable Backup

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Masking Failures from Application Performance in Data Center Networks with Shareable Backup DingmingWu+,Yiting Xia+*, XiaoyeStevenSun+, XinSunny Huang+,SimbarasheDzinamarira+, T. S. Eugene Ng+ +Rice University, *Facebook, Inc.

  2. Data Center Network Should be Reliable but…

  3. NetworkFailuresareDisruptive • Median case of failures: 10% less traffic delivered • Worst 20% of failures: 40% less traffic delivered Gill et al. SIGCOMM 2011

  4. Today’sFailureHandling---Rerouting • Fast local rerouting  inflatedpathlength • Global optimal rerouting  highlatencyofroutesupdates • Impact flows not traveling trough the failure location

  5. Impact on Coflow Completion Time (CCT) • Facebookcoflowtrace • k=16Fat-treenetwork • Globaloptimal rerouting

  6. DoWeHaveOther Options? • Restores network capacity immediately after failure • Be cost efficient • --Small pool of backup switch • How do we achieve that?

  7. Circuit Switches • Physicallayerdevice • Circuitcontrolledbysoftware C A • Examples • --optical 2D-MEMS switch, 40us, $10 per-port cost • --electrical cross-point switch, 70ns, $3 per-port cost B D

  8. IdealArchitecture Circuit Switch … … … BackupSwitch Servers Regularswitches • Entirenetworksharesonebackupswitch • Unreasonablehighport-countofcircuitswitch • Replaceanyfailedswitchwhennecessary • Singlepointoffailure

  9. How to Make It Practical • Feasibility • -small port-count circuit switches • Scalability • -partition network into failure groups • -distribute circuit switches across the network • Low cost • -small backup pool • -share backup switches per failure groups

  10. ShareBackupArchitecture AnoriginalFat-treewith k=6 • Partitiontheswitchesintofailuregroups;eachwithk/2switches. Corelayer • Addbackupswitchesperfailuregroups Agg.layer Edgelayer

  11. EdgeLayer Edge switches Backup Switch 0 1 2 Circuit switches 1 0 2 0 2 1 Servers i

  12. AggregationLayer Backup switch Agg. switches 0 1 2 ? 1 0 2 1 2 Circuit switches 0 1 2 0 1 2 0 ? Edge switches Backup switch 0 1 2

  13. Core Layer Core switches 0 3 6 1 4 7 2 5 8 Circuit switches Aggregation switches Backup switch 0 1 2 0 1 2 0 1 2

  14. Recover First, Diagnose Later • FailureRecovery • --switchfailurereplacedbybackupsviacircuitreconfiguration • --linkfailureswitchesonbothsidearereplaced • Automatic failure diagnosis performed offline • -details in the paper

  15. Live Impersonation of Failed Switch Backup switch Edge switches 0 1 2 Routing Table of Every Edge Switch Routing Table 0 VLAN 0 Routing Table 1 VLAN 1 Routing Table 2 VLAN 2 Servers

  16. Live Impersonation of Failed Switch Backup switch Edge switches 0 1 2 Routing Table of Every Edge Switch Routing Table 0 VLAN 0 Routing Table 1 VLAN 1 Routing Table 2 VLAN 2 0 Servers

  17. Live Impersonation of Failed Switch Edge switches Backup switch 0 1 2 Routing Table of Every Edge Switch Routing Table 0 VLAN 0 Routing Table 1 VLAN 1 Routing Table 2 VLAN 2 0 Servers

  18. What does control system do? • Collects keep-alive messages & link status reports from switches • Reconfigures circuit switches under failures • Performs offline failure diagnosis • Implications • -needs to talk to many circuit switches and packet switches • -keeps a large amount of states of circuit/switch/link status

  19. DistributedControl System • Onecontrollerforafailuregroupofk/2switches --configuresthecircuitswitchesadjacent toswitchesinthegroup • Maintainsonlylocalcircuitconfigurationsinitsgroup • --doesnotsharestateswithothercontrollers • Talkstocircuitswitchesusinganout-of-bandcontrolnetwork

  20. Summary • FastFailureRecovery • --asfastastheunderlyingcircuitswitchingtechnology • LiveImpersonation • --Traffic is redirected to the backups in physical layer • --Switchesinafailuregrouphavesameroutingtables,useVLANidfordifferentiation • --Regular switches recovered from failures become backup switchesthemselves Fastfailurerecovery,nopathdilation,noroutingdisturbance

  21. Evaluation • Bandwidth Advantage • --Iperf throughput on testbed • Application performance • --MapReduce job completion time

  22. Bandwidth Advantage • 4racks,8 servers,12switches • 8 iPerf flows saturate the network core ShareBackup restores network to full capacity regardlessoffailurelocations

  23. Application Performance 1.2X MapReduce Sort w/ 100GB input data 4.2X ShareBackup preservesapplicationperformanceunderfailures!

  24. ExtraCost • Smallport-countcircuitswitches---veryinexpensive • --e.g.$3per-portcostforcross-pointswitches • Smallbackupswitchpool • --1backupperfailuregroupisusuallyenough • --k = 48 fat-tree with 27648 servers ~6.7%extranetworkcost • Partialdeployment • --failuresmoredestructiveatedgelayer • --employbackuponlyforToRfailures

  25. Conclusion • ShareBackup:anarchitecturalsolutionforfailurerecoveryinDCNs • --usescircuitswitchingforfastfailover • --is aneconomicalapproachofusingbackupsinnetworks • --preservesapplicationperformanceunderfailures • Keytakeaways: • --reroutingisnotthe only approach forfailurerecovery • --fast,transparentfailurerecoveryispossiblethroughcarefulbackupplacements&fastcircuitswitching

  26. Backup---ControlSystemFailures • Circuitswitchsoftwarefailure/controlchannelfailure • --circuitswitchesbecomeunresponsive • --keepexistingcircuitconfigurations,dataplaneisnotimpacted • --fallbacktorerouting • Hardware/powerfailure • --controllerwillreceivelotsfailurereportsinashorttime • --callforhumanintervention • Controllerfailure • --state replication on shadow controllers

  27. Backup---Offline Failure Diagnosis 0 0 0 Aggregation switch ? ? • Recycle healthy switch - Only one switch has failed - Back to normal after reboot • Chain up circuit switches using side ports Circuit switches 0 0 0 ? ? Edge switches 17

  28. Backup---Offline Failure Diagnosis 0 0 0 Aggregation switch Circuit switches 0 0 0 Edge switches 18

More Related