1. Lecture 06 Distributed File Systems §6.1 Architecture §6.2 Process and Communication §6.3 Naming §6.4 Synchronization §6.5 Consistency and Replication §6.6 Fault Tolerance and Security

2. What’s Distributed File System? • A file system distributed across multiple nodes • To share data among processes • Persistent, secure, and reliable • Access using file system interfaces • A fundamental component of distributed systems • Typical systems: • NFS, AFS, etc.: earlier, typically with a single server • HDFS, GFS: newer, used in data centers • NFS, Network File System • UNIX-based, by Sun

3. §6.1 Architecture • Client-Server • Cluster based Client-Server • Symmetric, fully decentralized

4. Client-Server Architecture

5. Client-Server Architecture

6. Client-Server Architecture

7. File Operations of NFS

8. Cluster based Client-Server Architecture • Basically client-server • With a cluster of file servers • For large scale, parallel applications • Two further types • HPC cluster • Data center cluster

9. DFS of HPC Clusters • Small to middle size • File striping to realize parallel access b2 b1 b3 a1 a3 a2 c1 c2 c3

10. DFS of Data Center Clusters • Very large scale: tens of thousands PCs • Large file of small data pieces, usually appending • Frequent faults a1 a1 a2 a3

11. Symmetric Architecture • P2P , high availability and scalability • Ivy as an example

12. §6.2 Process and Communication • NFS: client and server processes • One of key issue in design: stateness • From stateless (v3) to stateful (v4) • Stateless • Simple, especially in recovery • Less powerful, e.g. access authentication, locking a file are not easy • Stateful • Complex • More powerful: supporting WAN, efficient caching, callback to client, etc.

13. Process Communication in NFS • Open Network Computing RPC (ONC RPC) • To make NFS independent from underlying OS, networks, and transport protocols. • Every NFS operation can be implemented as a single RPC to a file server. • Client is responsible for organize complex operations • To make the sever simple

14. Remote Procedure Calls in NFS Figure 11-7. (a) Reading data from a file in NFS version 3. (b) Reading data using a compound procedure in version 4.

15. The RPC2 Subsystem in Coda • Figure 11-8. Side effects in Coda’s RPC2 system.

16. The RPC2 Subsystem in Coda • Figure 11-9. Local copy invalidation via multicasting

17. §6.3 Naming • Naming in NFS • Name space is local Figure 11-11. Mounting (part of) a remote file system in NFS.

18. Naming in NFS • Pros: • Transparency, simplicity • Cons: • Hard to share files among clients • Names are locally valid • Solution: standardize the major directories

19. Naming in NFS Figure 11-12. Mounting nested directories from multiple servers in NFS.

20. Automounting • To mount directories transparently • Automounter will cause performance degradation

21. Automounting via Symbolic Linking • To limit the use of automounter in a special directory

22. Naming in Global Name Space • To share files with a globally unified name space • Via ftp protocol • not convenient for using • Dedicatedly designed wide-area file system • Modifications to OS kernel • too costly, constrained • Global Name Space Service (GNS) • To integrate existing file systems into a single, global name space • Using only user-level solutions

23. GNS • To merge name space only • Not to provide interfaces to access files • Basic idea • a GNS client maintains a virtual tree, where • each node is either a directory or a junction. • a junction is a special node that indicates that name resolution is to be taken over by another process.

24. GNS • Five types of junctions in GNS

25. §6.4 Synchronization • To control concurrent accessing to shared files • What results are “correct”? – semantics • How to realize? – usually based on “locking”

26. Semantics of File Sharing

27. The UNIX semantics • For single machine • Absolute time ordering • “Read the last write”

28. Semantics for Distributed Systems • In distributed systems • Multiple machines • No global time • Single file server • Easy to achieve UNIX semantics • Multiple servers, or • Single server wt client caches • Reading old values may occur

29. Session Semantics • Changes to an open file are initially visible only to the process (or possibly machine) that modified the file. • Only when the file is closed are the changes made visible to other processes (or machines). • Adopted by most DFSs • What happens when two nodes cache and modify the same file simultaneously? • Determine based on the “close” time, or • Leave undefined

30. Immutable Files • A special approach • All files are immutable • No way to open a file for writing • Only operations on files are “create” and “read” • Update a file by “replacing” it • Then, no simultaneous writing at all • How about “simultaneous” replacing? • Handled similar as in session semantics • How about to replace a file being read? • Allow reading to continue, or don’t allow.

31. Transaction based File Sharing • File operations are included in transactions • Atomic and consistent

32. File Locking • A central lock manager, in general • Locking is complex, to allow concurrent accessing • Different locks exist, and • Different granularities used • NFS v4 as an example

33. File Locking in NFS v4 • Conceptually simple • If a “lock” fails due to confliction, an error message is returned (nonblocking),and then • The client tries later (simple) or • Request the server to put its request on a FIFO list, and poll the server in time (fairness guaranteed)

34. File Share Reservation in NFS • An implicit way to lock a file • Completely independent from locking management • Specifying the type of access upon opening a file

35. §6.5 Consistency and Replication • Cache, client side • Replication, server side • NFS as the example

36. Client-side Caching in NFS • v3: not considered in NFS protocol itself • Handled in an implementation-dependent way • Can be stale for a few seconds or even 30 seconds • No consistency guaranteed

37. Client-side Caching in NFS • v4: considered caching • Still in an implementation-dependent way • Session semantics specified • When a client opens a file and caches the data it obtains from the server as the result of various read operations. • Write operations can be carried out in the cache as well. • Flushing cache back to the server upon closing the file • Clients at the same machine can share cache • Cache can be kept after file close • Need to be revalidated after re-opening

38. Open Delegation • A server may delegate some of its rights to a client when a file is opened for writing • The client machine is allowed to locally handle open and close operations from other clients on the same machine • file locking requests can be handled locally. • The server will still handle locking requests from clients on other machines, by simply denying those clients access to the file.

39. Open Delegation • The server needs to be able to recall the delegation • Recalling a delegation is done by a callback to the client

40. Server-side Replication • Replicating files near the sever or across different servers • Less popular than caching • Mostly for fault tolerance rather than performance (Geo-replication across data centers are popular)

41. §6.6 Fault Tolerance and Security • Replication over fault-tolerant server groups is the most popular technique • Also some other special issues, e.g. Byzantine failures, are handled specially.

42. Security in Distributed File Systems • Centralized, usually to have the servers handle authentication and access control • As in NFS • Authentication by a separate service, such as Kerberos • Authorization (access control) by the file servers • Key drawback: poor scalability • Decentralized, good scalability

43. Security in NFS • Mainly by secure communications (RPCs).

44. Authentication Ways in NFS • System authentication • A client simply passes its user ID and group ID to the server, in plaintext • No authentication in fact. • Diffie-Hellman key exchange • a public-key cryptosystem to establish a session key • in old NFS • Kerberos • Better and popular

45. Kerberos based Secure RPC in NFS • RPCSEC_GSS: a general security framework that can support different security mechanisms for setting up secure channels • LIPKEY is a public-key system that allows clients to be authenticated using a password while servers can be authenticated using a public key.

46. Kerberos • Basic idea: shared secrets • User prove to KDC who he is, by log on; • KDC generates shared secret between client and file server KDC ticket server generates S T “Need to access fs” file server Kclient[S] Kfs[S] client S: specific to {client, fs} pair; “short-term session-key”; has expiration time (e.g. 8 hours);

47. Kerberos Interactions KDC “Need to access fs” ticket server generates S 1. T Kclient[S], ticket = Kfs[ use S for client] client ticket=Kfs[use S for client], S[client, time] 2. S{time} client file server • why “time”: guard against replay attack • mutual authentication • File server doesn’t store S, which is specific to {client, fs} • Client doesn’t contact “ticket server” every time it contacts fs

48. Access Control (Authorization) in NFS • ACL file attribute • A list of access control entries • Each entry specifies the access rights for a specific user or group, e.g.

49. Access Control in NFS • Figure 11-30. The various kinds of users and processes distinguished by NFS with respect to access control.

50. Decentralized Authentication • Secure File Systems (SFS) in combination with decentralized authentication servers • The basic idea is simple • Allow a a user to specify that a remote user has certain privileges on his files • For example, Alice can specify that "Bob, whose details can be found at X," has certain privileges. • The authentication server that handles Alice's credentials could then contact server X to get information on Bob. • Users must be globally known to all authentication servers.