460 likes | 710 Vues
Chord+DHash+Ivy: Building Principled Peer-to-Peer Systems. Robert Morris rtm@lcs.mit.edu Joint work with F. Kaashoek, D. Karger, I. Stoica, H. Balakrishnan, F. Dabek, T. Gil, B. Chen, and A. Muthitacharoen. What is a P2P system?. Node. Node. Node. A distributed system architecture:
E N D
Chord+DHash+Ivy: Building Principled Peer-to-Peer Systems Robert Morris rtm@lcs.mit.edu Joint work with F. Kaashoek, D. Karger, I. Stoica, H. Balakrishnan, F. Dabek, T. Gil, B. Chen, and A. Muthitacharoen
What is a P2P system? Node Node Node • A distributed system architecture: • No centralized control • Nodes are symmetric in function • Large number of unreliable nodes • Enabled by technology improvements Internet Node Node
The promise of P2P computing • High capacity through parallelism: • Many disks • Many network connections • Many CPUs • Reliability: • Many replicas • Geographic distribution • Automatic configuration • Useful in public and proprietary settings
…. node node node Distributed hash table (DHT) (Ivy) Distributed application data get (key) put(key, data) (DHash) Distributed hash table lookup(key) node IP address (Chord) Lookup service • Application may be distributed over many nodes • DHT distributes data storage over many nodes
A DHT has a good interface • Put(key, value) and get(key) value • Call a key/value pair a “block” • API supports a wide range of applications • DHT imposes no structure/meaning on keys • Key/value pairs are persistent and global • Can store keys in other DHT values • And thus build complex data structures
A DHT makes a good shared infrastructure • Many applications can share one DHT service • Much as applications share the Internet • Eases deployment of new applications • Pools resources from many participants • Efficient due to statistical multiplexing • Fault-tolerant due to geographic distribution
Many recent DHT-based projects • File sharing [CFS, OceanStore, PAST, …] • Web cache [Squirrel, ..] • Backup store [Pastiche] • Censor-resistant stores [Eternity, FreeNet,..] • DB query and indexing [Hellerstein, …] • Event notification [Scribe] • Naming systems [ChordDNS, Twine, ..] • Communication primitives [I3, …] Common thread: data is location-independent
The lookup problem N2 N1 N3 Put (Key=“title” Value=file data…) Internet ? Client Publisher Get(key=“title”) N4 N6 N5 • At the heart of all DHTs
Centralized lookup (Napster) N2 N1 SetLoc(“title”, N4) N3 Client DB N4 Publisher@ Lookup(“title”) Key=“title” Value=file data… N8 N9 N7 N6 Simple, but O(N) state and a single point of failure
Flooded queries (Gnutella) N2 N1 Lookup(“title”) N3 Client N4 Publisher@ Key=“title” Value=file data… N6 N8 N7 N9 Robust, but worst case O(N) messages per lookup
Routed queries (Freenet, Chord, etc.) N2 N1 N3 Client N4 Lookup(“title”) Publisher Key=“title” Value=file data… N6 N8 N7 N9
Chord lookup algorithm properties • Interface: lookup(key) IP address • Efficient: O(log N) messages per lookup • N is the total number of servers • Scalable: O(log N) state per node • Robust: survives massive failures • Simple to analyze
Chord IDs • Key identifier = SHA-1(key) • Node identifier = SHA-1(IP address) • SHA-1 distributes both uniformly • How to map key IDs to node IDs?
Chord Hashes a Key to its Successor Key ID Node ID N10 K5, K10 K100 N100 Circular ID Space N32 K11, K30 K65, K70 N80 N60 K33, K40, K52 • Successor: node with next highest ID
Basic Lookup N5 N10 N110 “Where is key 50?” N20 N99 “Key 50 is At N60” N32 N40 N80 N60 • Lookups find the ID’s predecessor • Correct if successors are correct
Successor Lists Ensure Robust Lookup 10, 20, 32 N5 20, 32, 40 N10 5, 10, 20 N110 32, 40, 60 N20 110, 5, 10 N99 40, 60, 80 N32 N40 60, 80, 99 99, 110, 5 N80 N60 80, 99, 110 • Each node remembers r successors • Lookup can skip over dead nodes to find blocks
Chord “Finger Table” Accelerates Lookups ½ ¼ 1/8 1/16 1/32 1/64 1/128 N80
Chord lookups take O(log N) hops N5 N10 N110 K19 N20 N99 N32 Lookup(K19) N80 N60
Simulation Results: ½ log2(N) Average Messages per Lookup Number of Nodes • Error bars mark 1st and 99th percentiles
DHash Properties • Builds key/value storage on Chord • Replicates blocks for availability • Caches blocks for load balance • Authenticates block contents
DHash Replicates blocks at r successors N5 N10 N110 N20 N99 Block 17 N40 N50 N80 N68 N60 • Replicas are easy to find if successor fails • Hashed node IDs ensure independent failure
DHash Data Authentication • Two types of DHash blocks: • Content-hash: key = SHA-1(data) • Public-key: key is a public key, data is signed by that key • DHash servers verify before accepting • Clients verify result of get(key)
Ivy File System Properties • Traditional file-system interface (almost) • Read/write for multiple users • No central components • Trusted service from untrusted components
Straw Man: Shared Structure Root Inode • Standard meta-data in DHT blocks? • What about locking during updates? • Requires 100% trust Directory Block File1 Inode File2 Inode File3 Inode File3 Data
Ivy Design Overview • Log structured • Avoids in-place updates • Each participant writes only its own log • Avoids concurrent updates to DHT data • Each participant reads all logs • Private snapshots for speed
Internet Ivy Software Structure DHT Node user Ivy Server App system calls DHT Node NFS RPCs NFS Client DHT Node kernel
One Participant’s Ivy Log Record 1 Record 2 Record 3 Log Head Immutable content-hash DHT blocks Mutable public-key signed DHT block • Log-head contains DHT key of most recent record • Each record contains DHT key of previous record
Ivy I-Numbers • Every file has a unique I-Number • Log records contain I-Numbers • Ivy returns I-Numbers to NFS client • NFS requests contain I-Numbers • In the NFS file handle
NFS/Ivy Communication Example Local NFS Client Local Ivy Server LOOKUP(“d”, I-Num=1000) I-Num=1000 CREATE(“aaa”, I-Num=1000) I-Num=9956 WRITE(“hello”, 0, I-Num=9956) OK • echo hello > d/aaa • LOOKUP finds the I-Number of directory “d” • CREATE creates file “aaa” in directory “d” • WRITE writes “hello” at offset 0 in file “aaa”
Log Records for File Creation Type: Create I-num: 9956 Type: Link Dir I-num: 1000 File I-num: 9956 Name: “aaa” Type: Write I-num: 9956 Offset: 0 Data: “hello” … Log Head • A log record describes a change to the file system
Scanning an Ivy Log Type: Link Dir I-num: 1000 File I-num: 9956 Name: “aaa” Type: Link Dir I-num: 1000 File I-num: 9876 Name: “bbb” Type: Remove Dir I-num: 1000 Name: “aaa” • A scan follows the log backwards in time • LOOKUP(name, dir I-num): last Link, but stop at Remove • READDIR(dir I-num): accumulate Links, minus Removes
Finding Other Logs: The View Block Log Head 1 View Block Pub Key 1 Pub Key 2 Pub Key 3 Log Head 2 Log Head 3 • View block is immutable (content-hash DHT block) • View block’s DHT key names the file system • Example: /ivy/37ae5ff901/aaa
Reading Multiple Logs 27 31 32 20 Log Head 1 26 27 28 29 30 Log Head 2 • Problem: how to interleave log records? • Red numbers indicate real time of record creation • But we cannot count on synchronized clocks
Vector Timestamps Encode Partial Order 27 31 32 20 Log Head 1 26 27 28 29 30 Log Head 2 • Each log record contains vector of DHT keys • One vector entry per log • Entry points to log’s most recent record
Snapshots • Scanning the logs is slow • Each participant keeps a private snapshot • Log pointers as of snapshot creation time • Table of all I-nodes • Each file’s attributes and contents • Reflects all participants’ logs • Participant updates periodically from logs • All snapshots share storage in the DHT
Simultaneous Updates • Ordinary file servers serialize all updates • Ivy does not • Most cases are not a problem: • Simultaneous writes to the same file • Simultaneous creation of different files in same directory • Problem case: • Unlink(“a”) and rename(“a”, “b”) at same time • Ivy correctly lets only one take effect • But it may return “success” status for both
Integrity • Can attacker corrupt my files? • Not unless attacker is in my Ivy view • What if a participant goes bad? • Others can ignore participant’s whole log • Ignore entries after some date • Ignore just harmful records
Ivy Performance • Half as fast as NFS on LAN and WAN • Scalable w/ # of participants • These results were taken yesterday…
Ivy Server DHash Server App NFS Client Local Benchmark Configuration • One log • One DHash server • Ivy+DHash all on one host
Ivy Local Performance on MAB • Modified Andrew Benchmark times (seconds) • NFS: client – LAN – server • 7 seconds doing public key signatures, 3 in DHash
WAN Benchmark Details • 4 DHash nodes at MIT, CMU, NYU, Cornell • Round-trip times: 8, 14, 22 milliseconds • No DHash replication • 4 logs • One active writer at MIT • Whole-file read on open() • Whole-file write on close() • NFS client/server round-trip time is 14 ms
Ivy WAN Performance • 47 seconds fetching log heads, 4 writing log head, • 16 inserting log records, 22 in crypto and CPU
Ivy Performance w/ Many Logs • MAB on 4-node WAN • One active writer • Increasing cost due to growing vector timestamps
Related Work • DHTs • Pastry, CAN, Tapestry • File systems • LFS, Zebra, xFS • Byzantine agreement • BFS, OceanStore, Farsite
Summary • Exploring use of DHTs as a building block • Put/get API is general • Provides availability, authentication • Harnesses decentralized peer-to-peer groups • Case study of DHT use: Ivy • Read/write peer-to-peer file system • Trustable system from untrusted pieces http://pdos.lcs.mit.edu/chord