Farsite: Ferderated, Available, and Reliable Storage for an Incompletely Trusted Environment

1. Farsite: Ferderated, Available, and Reliable Storage for an Incompletely Trusted Environment Microsoft Reseach, Appear in OSDI’02

2. Design Assumption • 100,000 machines in a large corporation or university, interconnected by a high-bandwidth, low-latency network • Allow large-scale read-only sharing • Allow small-scale read/write sharing • A small fraction of users misbehave

3. Enabling Technology Trends • Large amount of unused disk space enables the use of replication for reliability • Relatively low cost of strong cryptography enables distributed security

4. Problems • Namespace roots • A file system is a hierachical directory namespace, originated at a root • Allows multiple roots, each of which can be regarded as a virtual file server • A root cooresponds to a set of participating machines • Trust and Certification • The security of any distributed system is an issue of trust • Manage trust using public-key-cryptographic certificates • A namespace certificate • A user certificate • A machine certificate

5. Basic System • Each machine performs three roles: a client, a member of a directory group, and a file host • A directory group: a set of machine that collectively manage file information using a Byzantine-fault-tolerant protocol • A file host: a machine used to store file data replicas

6. Performance Considerations • Problems ? • All FS metadata operations involve Byzantine-fault-tolerant protocol(BFT) • BFT is high-cost • Solution • Local caching improves read performance (by content leases) • Batch logged updates(write-back caching, due to many writes are deleted or overwritten shortly after they occur)

7. Security • Access control by ACL • Privacy • Convergent encryption to protect the file data • Exclusive encryption to protect directory or file names • Integrity by a Merkle hash tree

8. Scalability • When a directory group becomes overloaded, it can delegate part of its namespace to another group • When open a file/directory with a paticular pathname, it needs to determine which group of machines is responsible for that name • Hint-based pathname translation (caching) like in Sprite

9. Taming aggressive replication in the Pangaea wide-area file system HP Labs

10. Design Goals • Speed: hide the wide-area networking latency • Availability and autonomy • Network economy: transfer data between nodes in physical proximity, thereby reducing latency and bandwidth

11. Structure of a file system • Gold replicas • The directory entry of a file lists the file’s gold replicas • Form a clique • Bronze replicas

12. Replica set management • Pervasive replication: a replica is created whenever a file is accessed by a user • File creation • Replica addition: the new replica S must be added to the graph (m edged) • adds an edge to a random gold replica (from a different region than S) • Asks a random gold replica P, to pick the replica (among P’s immediate graph neighbors)closest to S • Asks P to choose m-2 random replicas using random walk • Name-space containment

13. Propagating updates • Efficient and reliable update propagation • Delta propagation, harbingers, and using a spanning tree to exploit physical topology • Conflict resolution: combing version vectors and last-writer-win rules • Lack of strong consistency guarantees: eventually achieved

14. Questions? • Graph-based replica for each file, too much metadata to maintain • Like a multicast-based file system, updates are propagated using multicast

15. Discussion • Metadata and data management in a distributed file sytem • Either mutable, but have to trust some machines, like xFS, or Farsite using Byzantine-fault-tolerant to trust part of machines to serialize updates • Or immutable, using logged updates, it relies on each individual user to form the image of a file system • The replication factor of metadata and data maybe differ according to their usage?