Distributed Filesystems: NFS and GFS

1. Distributed Filesystems:NFS and GFS Costin Raiciu Advanced Topics in Distributed Systems 18thOctober, 2011 Slides courtesy of Brad Karp (UCL) and Robert Morris (MIT)

2. Distributed Filesystems • Why might we need them? • Sharing data: many users read/write the same files • Files too big to store on a single machine • Better fault tolerance – via replication across machines • Better performance – if reads are serviced by many replicas

3. What defines a Distributed FS? • Need some common namespace (dir/open/close) • Metadata • which files are in this directory? • where are the contents of this file stored? • This is typically small in size • We care a lot about consistency • Need a way to access data (read/write) • There will be lots of data

4. How do we design a distributed filesystem? • Should we use a centralized server? • Easier to manage • All clients connect to it • No fault tolerance (but can be added) • Or should we use many servers? • Who holds metadata? • Who holds data?

5. Are we allowed to change apps? • It influences the design quite a bit! • Apps unchanged • Great deployability • Bound to Unix local filesystem semantics • Apps changed • We can do whatever we want • But optimal design depends on app!

6. Network File System

7. NFS Is Relevant • Original paper from 1985 • Very successful, still widely used today • Early result; much subsequent research in networked filesystems “fixing shortcomings of NFS”

8. Why Did They Build NFS? • Sharing data: many users reading/writing same files but running on separate machines • Manageability: ease of backing up one server • Disks may be expensive (true when NFS built; no longer true) • Diskless workstations

9. Goals for NFS • Easily deployed • Easy to add to existing UNIX systems • Work with existing, unmodified apps: • Same semantics as local UNIX filesystem • Compatible with non-UNIX OSes • Wire protocol cannot be too UNIX-specific • Efficient “enough” • Needn’t offer same performance as local UNIX filesystem

10. NFS Architecture App1 App2 Clients LAN Server (w/disk) syscalls User Kernel Filesystem RPCs

11. Simple Example: Reading a File • What RPCs would we expect for: fd = open(“f”, 0); read(fd, buf, 8192); close(fd);

12. Simple Example: NFS RPCs for Reading a File • Where are RPCs for close()?

13. NFS Server is stateless • Simplifies crash recovery • Allows it to scale to many concurrent clients • No need to remember them! • NFS runs mainly on top of UDP

14. File Handle: Function and Contents • 32-byte name, opaque to client • Identifies object on remote server • Must be included in all NFS RPCs • File handle contains: • filesystem ID • i-number (essentially, physical block ID on disk) • generation number

15. Generation Number: Motivation • Client 1 opens file • Client 2 opens same file • Client 1 deletes the file, creates new one • UNIX local filesystem semantics: • Client 2 (App 2) sees old file • In NFS, suppose server re-uses i-node • Same i-number for new file as old • RPCs from client 2 refer to new file’s i-number • Client 2 sees new file!

16. Trade precise UNIX fs semantics for simplicity Generation Number: Solution • Each time server frees i-node, increments its generation number • Client 2’s RPCs now use old file handle • Server can distinguish requests for old vs. new file • Semantics still not same as local UNIX fs! • Apps 1 and 2 sharing local fs: client 2 will see old file • Clients 1 and 2 on different workstations sharing NFS fs: client 2 gets error “stale file handle”

17. Why i-numbers, not Filenames? • Local UNIX fs: client 1 reads dir2/f • NFS with pathnames: client 1 reads dir1/f • Concurrent access by clients can change object referred to by filename • Why not a problem in local UNIX fs? • i-number refers to actual object, not filename

18. Where Does Client Learn File Handles? • Before READ, client obtains file handle using LOOKUP or CREATE • Client stores returned file handle in vnode • Client’s file descriptor refers to vnode • Where does client get very first file handle?

19. vnode’s purpose: remember file handles! NFS Implementation Layering • Why not just send syscalls over wire? • UNIX semantics defined in terms of files, not just filenames: file’s identity is i-number on disk • Even after rename, all these refer to same object as before: • File descriptor • Home directory • Cache contents

20. Example: Creating a File over NFS • Suppose client does: fd = creat(“d/f”, 0666); write(fd, “foo”, 3); close(fd); • RPCs sent by client: • newfh = LOOKUP (fh, “d”) • filefh = CREATE (newfh, “f”, 0666) • WRITE (filefh, 0, 3, “foo”)

21. Server Crashes and Robustness • Suppose server crashes and reboots • Will client requests still work? • Will client’s file handles still make sense? • Yes! File handle is disk address of i-node • What if server crashes just after client sends an RPC? • Before server replies: client doesn’t get reply, retries • What if server crashes just after replying to WRITE RPC?

22. WRITE RPCs and Crash Robustness • What must server do to ensure correct behavior when crash after WRITE from client? • Client’s data safe on disk • i-node with new block number and new length safe on disk • Indirect block safe on disk • Three writes, three seeks: 45 ms • 22 WRITEs/s, so 180 KB/s

23. WRITEs and Throughput • Design for higher write throughput: • Client writes entire file sequentially at Ethernet speed (few MB/s) • Update inode, &c. afterwards • Why doesn’t NFS use this approach? • What happens if server crashes and reboots? • Does client believe write completed? • Improved in NFSv3: WRITEsasync, COMMIT on close()

24. Client Caches in NFS • Server caches disk blocks • Client caches file content blocks, some clean, some dirty • Client caches file attributes • Client caches name-to-file-handle mappings • Client caches directory contents • General concern: what if client A caches data, but client B changes it?

25. Multi-Client Consistency • Real-world examples of data cached on one host, changed on another: • Save in emacs on one host, “make” on other host • “make” on one host, run program on other host • (No problem if users all run on one workstation, or don’t share files)

26. Consistency Protocol: First Try • On every read(), client asks server whether file has changed • if not, use cached data for file • if so, issue READ RPCs to get fresh data from server • Is this protocol sufficient to make each read() see latest write()? • What’s effect on performance? • Do we need such strong consistency?

27. Compromise:Close-to-Open Consistency • Implemented by most NFS clients • Contract: • if client A write()s a file, then close()s it, • then client B open()s the file, and read()s it, • client B’s reads will reflect client A’s writes • Benefit: clients need only contact server during open() and close()—not on every read() and write()

28. Fixes “emacs save, then make” example… …so long as user waits until emacs says it’s done saving file! Compromise:Close-to-Open Consistency

29. Close-to-Open Implementation • FreeBSD UNIX client (not part of protocol spec): • Client keeps file mtime and size for each cached file block • close() starts WRITEs for all file’s dirty blocks • close() waits for all of server’s replies to those WRITEs • open() always sends GETATTR to check file’s mtime and size, caches file attributes • read() uses cached blocks only if mtime/length have not changed • client checks cached directory contents with GETATTR and ctime

30. Name Caching in Practice • Name-to-file-handle cache not always checked for consistency on each LOOKUP • If file deleted, may get “stale file handle” error from server • If file renamed and new file created with same name, may even get wrong file’s contents

31. IP-address-based authentication of mount requests weak at best; no auth of server to client Security not a first-order goal in original NFS NFS: Secure? • What prevents unauthorized users from issuing RPCs to an NFS server? • e.g., remove files, overwrite data, &c. • What prevents unauthorized users from forging NFS replies to an NFS client? • e.g., return data other than on real server

32. Despite its limitations, NFS a huge success: Simple enough to build for many OSes Correct enough and performs well enough to be practically useful in deployment Limitations of NFS • Security: what if untrusted users can be root on client machines? • Scalability: how many clients can share one server? • Writes always go through to server • Some writes are to “private,” unshared files that are deleted soon after creation • Can you run NFS on a large, complex network? • Effects of latency? Packet loss? Bottlenecks?

33. GFS: The Google File System

34. Motivating Application: Google • Crawl the whole web • Store it all on “one big disk” • Process users’ searches on “one big CPU” • More storage, CPU required than one PC can offer • Custom parallel supercomputer: expensive (so much so, not really available today)

35. GFS: File system for sharing data on clusters, designed with Google’s application workload specifically in mind Cluster of PCs as Supercomputer • Lots of cheap PCs, each with disk and CPU • High aggregate storage capacity • Spread search processing across many CPUs • How to share data among PCs? • NFS: share fs from one server, many clients • Goal: mimic original UNIX local fs semantics • Compromise: close-to-open consistency (performance) • Fault tolerance? • Ivy: shared virtual memory (we will discuss later) • Fine-grained, relatively strong consistency at load/store level • Fault tolerance?

36. Google Platform Characteristics • 100s to 1000s of PCs in cluster • Cheap, commodity parts in PCs • Many modes of failure for each PC: • App bugs, OS bugs • Human error • Disk failure, memory failure, net failure, power supply failure • Connector failure • Monitoring, fault tolerance, auto-recovery essential

37. Google File System: Design Criteria • Detect, tolerate, recover from failures automatically • Large files, >= 100 MB in size • Large, streaming reads (>= 1 MB in size) • Read once • Large, sequential writes that append • Write once • Concurrent appends by multiple clients (e.g., producer-consumer queues) • Want atomicity for appends without synchronization overhead among clients

38. GFS: Architecture • One master server (state replicated on backups) • Many chunk servers (100s – 1000s) • Spread across racks; intra-rack b/w greater than inter-rack • Chunk: 64 MB portion of file, identified by 64-bit, globally unique ID • Many clients accessing same and different files stored on same cluster

39. GFS: Architecture (2)

40. Holds all metadata in RAM; very fast operations on file system metadata Master Server • Holds all metadata: • Namespace (directory hierarchy) • Access control information (per-file) • Mapping from files to chunks • Current locations of chunks (chunkservers) • Manages chunk leases to chunkservers • Garbage collects orphaned chunks • Migrates chunks between chunkservers

41. Chunkserver • Stores 64 MB file chunks on local disk using standard Linux filesystem, each with version number and checksum • Read/write requests specify chunk handle and byte range • Chunks replicated on configurable number of chunkservers (default: 3) • No caching of file data (beyond standard Linux buffer cache)

42. Client • Issues control (metadata) requests to master server • Issues data requests directly to chunkservers • Caches metadata • Does no caching of data • No consistency difficulties among clients • Streaming reads (read once) and append writes (write once) don’t benefit much from caching at client

43. Client API • Is GFS a filesystem in traditional sense? • Implemented in kernel, under vnode layer? • Mimics UNIX semantics? • No; a library apps can link in for storage access • API: • open, delete, read, write (as expected) • snapshot: quickly create copy of file • append: at least once, possibly with gaps and/or inconsistencies among clients

44. Client Read • Client sends master: • read(file name, chunk index) • Master’s reply: • chunk ID, chunk version number, locations of replicas • Client sends “closest” chunkserver w/replica: • read(chunk ID, byte range) • “Closest” determined by IP address on simple rack-based network topology • Chunkserver replies with data

45. Client Write • Some chunkserver is primary for each chunk • Master grants lease to primary (typically for 60 sec.) • Leases renewed using periodic heartbeat messages between master and chunkservers • Client asks server for primary and secondary replicas for each chunk • Client sends data to replicas in daisy chain • Pipelined: each replica forwards as it receives • Takes advantage of full-duplex Ethernet links

46. Client Write (2) • All replicas acknowledge data write to client • Client sends write request to primary • Primary assigns serial number to write request, providing ordering • Primary forwards write request with same serial number to secondaries • Secondaries all reply to primary after completing write • Primary replies to client

47. Client Write (3)

48. Client Record Append • Google uses large files as queues between multiple producers and consumers • Same control flow as for writes, except… • Client pushes data to replicas of last chunk of file • Client sends request to primary • Common case: request fits in current last chunk: • Primary appends data to own replica • Primary tells secondaries to do same at same byte offset in theirs • Primary replies with success to client

49. Client Record Append (2) • When data won’t fit in last chunk: • Primary fills current chunk with padding • Primary instructs other replicas to do same • Primary replies to client, “retry on next chunk” • If record append fails at any replica, client retries operation • So replicas of same chunk may contain different data—even duplicates of all or part of record data • What guarantee does GFS provide on success? • Data written at least once in atomic unit

50. GFS: Consistency Model • Changes to namespace (i.e., metadata) are atomic • Done by single master server! • Master uses log to define global total order of namespace-changing operations • Data changes more complicated • Consistent: file region all clients see as same, regardless of replicas they read from • Defined: after data mutation, file region that is consistent, and all clients see that entire mutation