Bigtable : A Distributed Storage System for Structured Data

1. Bigtable: A Distributed Storage System for Structured Data F. Chang, J. Dean, S. Ghemawat, W.C. Hsieh, D.A. WallachM. Burrows, T. Chandra, A. Fikes, R.E. Gruber Google, Inc. TianyangHU

2. Outline • Introduction • Data Model • Building Blocks • Implementation • Refinements • Performance Evaluation • Future Work

3. Motivation • Scalability • worldwide applications & users • huge amount of communication & data

4. Bigtable • Distributed storage system • petabytes of data, thousands of machines • simple data model with dynamic control • applicability, scalability, performance, availability • Used by more than 60 applications of Google

5. Outline • Introduction • Data Model • Building Blocks • Implementation • Refinements • Performance Evaluation • Future Work

6. Data Model – Overview • Sparse, distributed, persistent multidimensional sorted map.

7. Data Model – Example • “Webtable” stores copy of web pages & their related information. • row key: URL (reverse hostname) • column key: attribute name • timestamp: time that the page is fetched

8. Data Model – Rows • Row key: string (usually 10-100KB, max 64KB) • Every R/W of data under a single row key is atomic

9. Data Model – Rows • Sorted by row key in lexicographic order • Tablet: a certain range of rows • the unit of distribution & load balancing • good locality for data access

10. Data Model – Columns • Column families: group of column keys (same type) • the unit of access control • Column key: family:qualifier

11. Data Model – Timestamps • Timestamp: index multiple versions of the same data • not necessarily the “real time” • data clean up, garbage collection

12. Outline • Introduction • Data Model • Building Blocks • Implementation • Refinements • Performance Evaluation • Future Work

13. Building Blocks • SSTablefile format • persistent, ordered, immutable key-value (string-string) pairs • used internally to store Bigtabledata

14. Building Blocks • GFS • store log & data files • scalability, reliability, performance, fault tolerance • Chubby • a highly-available and persistent distributed lock service

15. Outline • Introduction • Data Model • Building Blocks • Implementation • Refinements • Performance Evaluation • Future Work

16. Bigtable Components • A library that is linked into every client • Many tablet servers • handle R/W to tablets with clients • One tablet master • assign tablets to tablet servers • detect addition & expiration of tablet servers • balance tablet-server load

17. Architecture

18. Tablet Location • Three-level hierarchy • root tablet (Only one, stores addresses of METADATA tablets) • METADATA tablets (stores addresses of user tablets) • user tablets

19. Tablet Location • Client caches (multiple) tablet locations • if the cache is stale, query again

20. Tablet Assignment • The tablet master uses Chubby to keeps track of • live tablet servers • each live tablet server acquires an exclusive lock on a corresponding file • tablet assignment status • compare tablets registered in METADATA tablet with tablets in tablet servers

21. Tablet Assignment • Case 1: some tablets are unassigned • master assigns them to tablet servers with sufficient room • Case 2: a tablet server stops its service • master detects it and assigns outstanding tablets to other servers. • Case 3: too many small tablets • master initiates merge • Case 4: a tablet grows too large • the corresponding tablet server initiates split and notifies master

22. Tablet Serving • A tablet is stored as a sequence of SSTables in GFS • Tablet mutations are logged in commit log • the “commit log” stores redo records • recent tablet versions are stored in memory (memtable) • older tablet versions are stored in GFS

23. Tablet Serving • Recover a tablet • 1. Tablet server fetches its metadata from METADATA tablet, which contains a list of SSTables that comprises a tablet and redo points. • 2. The server reads the indices of the SSTables into memory. • 3. The server applies all the mutations after the redo point.

24. Tablet Serving • Write operation on a tablet • 1. The tablet server checks the validity of the operation. • 2. The operation is logged in the commit log. • 3. Commit the operation. • 4. The content of tablet is inserted into memtable.

25. Tablet Serving • Read operation on a tablet • 1. The tablet server checks the validity of the operation. • 2. Execute the operation on a merged view of memtable & SSTables.

26. Compactions • Memtable grows as write operations execute • Two types of compactions • minor compaction • merging (major) compaction

27. Compactions • Minor compaction (when memtable size reaches a threshold) • 1. Freeze the memtable • 2. Create a new memtable • 3. Convert the memtable to an SSTable and write to GFS

28. Compactions • Merging compaction (periodically) • 1. Freeze the memtable • 2. Create a new memtable • 3. Merge a few SSTables & memtable into a new SSTable

29. Compactions • Major compaction • special case of merging compaction • merges allSSTables & memtable

30. Compactions • Why freeze & create memtable? • Incoming read and write operations can continue during compactions. • Advantages of compaction: • release the memory of the tablet server • reduce the amount of data that has to be read from the commit log during recovery if this tablet server dies

31. Outline • Introduction • Data Model • Building Blocks • Implementation • Refinements • Performance Evaluation • Future Work

32. Refinements • Locality groups • group multiple column families together • different locality groups are not typically accessed together • for each tablet, store each locality group in a separate SSTable • more efficient R/W

33. Refinements • Compression • similar data in same column, neighbouringrows, multiple versions • customized compression on SSTable block level (smallest component) • two-pass compression scheme • 1. Bentley and McIlroy’sscheme, compress long strings across a large window • 2. fast compression algorithm, look for repetitions in small window • experimental compression ratio: 10% (Gzip: 25-33%)

34. Refinements • Caching • two-level cache on tablet server • Scan cache (high-level): caches key-value pairs • Case: read the same data repeatedly • Block cache (low-level): caches SSTableblocks • Case: sequential read

35. Refinements • Commit-log implementation • one commit log per tablet incurs a large # of disk seeks • use single commit log for all tablets on a tablet server • benefits significantly during normal operation • complicates recovery • solution: sort the commit log entries first

36. Outline • Introduction • Data Model • Building Blocks • Implementation • Refinements • Performance Evaluation • Future Work

37. Performance Evaluation R/W rate per tablet server aggregate R/W rate

38. Outline • Introduction • Data Model • Building Blocks • Implementation • Refinements • Performance Evaluation • Future Work

39. Future Work • Resource sharing for different applications? • Hybrid with relational database? • complex query • security