1 / 50

NCCloud : A Network-Coding-Based Storage System in a Cloud-of-Clouds

NCCloud : A Network-Coding-Based Storage System in a Cloud-of-Clouds. Henry C. H. Chen Yuchong Hu Patrick P. C. Lee Yang Tang. IEEE Transactions on Computers, 15 August 2013. Outline. Introduction Repair in Multiple Cloud Storage FMSR Codes NCCloud Conclusion. Introduction.

vesta
Télécharger la présentation

NCCloud : A Network-Coding-Based Storage System in a Cloud-of-Clouds

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. NCCloud: A Network-Coding-Based Storage System in a Cloud-of-Clouds Henry C. H. Chen Yuchong Hu Patrick P. C. Lee Yang Tang IEEE Transactions on Computers, 15 August 2013

  2. Outline • Introduction • Repair in Multiple Cloud Storage • FMSR Codes • NCCloud • Conclusion

  3. Introduction • Cloud storage provides an on-demand remote backup solution. • A single cloud storage provider encounters the problem such as a single point of failure.

  4. Introduction • The general solution is to distribute data across different cloud providers. • stripe data • The fault-tolerance can be improved by the diversity of multiple clouds.

  5. Introduction-Data Failure • This paper focuses on unexpected permanent cloud failure. • a cloud fails permanently => activate repair. • maintain data redundancy and fault-tolerance. • A repair operation • retrieves data from existing surviving clouds. • reconstructs the lost data in a new cloud.

  6. Introduction-Data Failure • During repair, each surviving node • encode its stored data chunks. • send the encoded chunks to a new node • Regenerate the lost data.

  7. Introduction-Cost Problem • Today’s cloud storage providers charge users for outbound data. • While repairing failures, moving the enormous amount of data (repair traffic) can introduce significant monetary costs.

  8. Introduction-Repair Traffic Problem • In order to minimize repair traffic problem, regenerating codes [16] have been proposed. • store data redundantly in a distributed storage system. • require less repair traffic, but with the same fault-tolerance level. [16] Network Coding for Distributed Storage Systems

  9. Introduction-Regenerating Codes • But, most existing regenerating codes require storage nodes • equip with computation capabilities. • perform encoding operations during repair.

  10. Introduction-Regenerating Codes • In order to make regenerating codes portable to any cloud storage service. • This paper considers only a thin-cloud interface where storage nodes only support read/write.

  11. Introduction-NCCloud • In this paper, we present the design and implementation of NCCloud • a proxy-based storage system. • a fault-tolerant storage. • over multiple cloud storage providers.

  12. Introduction-FMSR • On top of NCCloud, we propose the functional minimum-storage regenerating (FMSR) codes. • The FMSR code implementation • maintain double-fault tolerance. • maintain the same storage cost as in RAID-6 • less repair traffic when recovering a single-cloud failure.

  13. Introduction-FMSR • FMSR codes are non-systematic • the encoded chunks was formed by linear combination of the original data chunks. • not keep the original data chunks as in systematic coding schemes.

  14. Outline • Introduction • Repair in Multiple Cloud Storage • FMSR Codes • NCCloud • Conclusion

  15. Repair in Multiple Cloud Storage • Transient failure • is short-term, such that the failed cloud will return to normal after some time and no outsourced data is lost.

  16. Repair in Multiple Cloud Storage • Permanent failure • is long-term, in the sense that the outsourced data on a failed cloud will become permanently unavailable. • example : • data center outages in disasters. • data loss and corruption. • malicious attacks.

  17. Outline • Introduction • Repair in Multiple Cloud Storage • FMSR Codes • Motivation • Implementation • NCCloud • Conclusion

  18. Motivation • This paper considers • distributed • multiple-cloud storage • data is striped • proxy-based design

  19. Motivation

  20. Fault-tolerant • Maximum Distance Separable property • (n, k)-MDS code • divide file into equal-size native chunks. • linearly combined to form code chunks. • distribute over n (larger than k) nodes. • reconstruct original file from any k of the n nodes. • tolerate the failures of any n − k nodes.

  21. Fault-tolerant • The FMSR codes can reconstruct the data of failed node from the surviving nodes. • download less data. • not reconstruct the whole file.

  22. Different Coding Schemes Storage size 2M Repair traffic M Storage size 2M Repair traffic 0.75M Storage size 2M Repair traffic 0.75M

  23. Double-fault Tolerant FMSR Codes • divide a file Minto 2(n − 2) native chunks. • generate 2n code chunks. • each node store two code chunks of size. • repair a failed node, repair traffic is . • RAID-6 codes, total storage size is , repair traffic is M. 50% saved

  24. Outline • Introduction • Repair in Multiple Cloud Storage • FMSR Codes • Motivation • Implementation • NCCloud • Conclusion

  25. FMSR Codes Implementation • FMSR codes do not require lost chunks to be exactly reconstructed • not identical to those in the failed node. • As long as the MDS property holds.

  26. FMSR Codes Implementation • This paper propose a two-phase checking scheme to ensure the code chunks on all nodes always satisfy the MDS property.

  27. FMSR Codes Implementation • Theimplementation assumes a thin-cloud interface. • File upload • File download • Repair

  28. File Upload • Native chunks : • Code chunks : • Encoding matrix of coefficients : • size • in the Galois field GF(pn)

  29. File Upload • Galois field GF(pn) Encoding coefficient vector

  30. File Download • Download the k(n−k) code chunks from any k of the n storage nodes. • The ECVs of the k(n−k) code chunks can form a k(n−k)×k(n−k) square matrix. • Obtain the original k(n − k) native chunks. • multiply the inverse of the square matrix with the code chunks.

  31. Iterative Repair • MDS property must hold even after iterative repairs. • This paper proposes a two-phase checking. • MDS property • rMDS property

  32. Satisfy MDS, but not rMDS

  33. Iterative Repair Step 1. Download the encoding matrix from a surviving node. Step 2. Select one ECV from each of the n-1 surviving nodes. Step 3. Generate a repair matrix . Step 4. Compute the ECVs for the new code chunks and reproduce a new encoding matrix.

  34. Iterative Repair Step 5. Given EM’, verify if those properties are satisfied. • verify MDS by enumerating all . • verify rMDS by n(n−k)n-1. • The corresponding encoding matrices must form a full rank. Step 6. Download the actual chunk data and regenerate new chunk data. • Step 4 : The new ECVs • Code chunks from surviving nodes

  35. rMDS Sustaining

  36. Time of Two-phase Checking

  37. Double-fault Tolerant Codes • Markov Model

  38. MTTDL, Compare to RAID-6 Mean Time To Data Loss

  39. Outline • Introduction • Repair in Multiple Cloud Storage • FMSR Codes • NCCloud • Conclusion

  40. NCCloud • A proxy that bridges user applications and multiple clouds. • Its design is built on three layers. • File system layer • Coding layer • Storage layer

  41. NCCloud • It is mainly implemented in Python, while the coding schemes are implemented in C for better efficiency.

  42. Goal of NCCloud • Compare the costs and response time of using RAID-6 and FMSR codes. • The cost advantage of FMSR over RAID-6, while maintaining acceptable response time.

  43. Goal of NCCloud • Normal operations • RAID-6 and FMSR incur similar storage costs. • Repair operation • FMSR save a significant amount of transfer costs over RAID-6.

  44. Cost Saving-Price

  45. Cost Saving • Normal operations • 1.25PB of data stored • FMSR : $86,851 monthly storage cost • RAID-6 : $86,851 monthly storage cost • Repair operation • RAID-6 : 1PB of data, $56,832 • FMSR : 0.5625PB of data, $33,894 Savingof $ 22,938

  46. Response Time-Local Cloud

  47. Response Time-Local Cloud

  48. Response Time-Commerical Cloud

  49. Outline • Introduction • Repair in Multiple Cloud Storage • FMSR Codes • NCCloud • Conclusion

  50. Conclusion • This paper present NCCloud providing the reliability of today’s cloud backup storage. • proxy-based • multiple-cloud storage system • NCCloud not only provides fault tolerance in storage, but also allows cost-effective repair. • The FMSR code implementation eliminates the encoding requirement of storage nodes during repair.

More Related