1 / 20

CSC590 Selected Topics

CSC590 Selected Topics. Bigtable: A Distributed Storage System for Structured Data Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber. by Haifa Alyahya 432920323. Outline. Introduction Data Model

oliver-russell
Télécharger la présentation

CSC590 Selected Topics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CSC590 Selected Topics Bigtable: A Distributed Storage System for Structured Data Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber byHaifa Alyahya 432920323

  2. Outline • Introduction • Data Model • APIs • Building Blocks • Implementation • Refinements • Performance • Real Applications • Conclusion

  3. Discussion • Bigtable(Bt) is a distributed storage system for managing structured data that is designed to scale to a very large size. • Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance.

  4. Introduction • Bigtable is designed to reliably scale to petabytes of data and thousands of machines. • Bigtable has achieved several goals: • Wide applicability. • Scalability. • High performance. • High availability.

  5. Motivation • Scale Problem • Lots of data • Millions of machines • Different project/applications • Hundreds of millions of users • Storage for (semi-)structured data. • No commercial system big enough • Couldn’t afford if there was one • Low-level storage optimization help performance significantly • Much harder to do when running on top of a database layer

  6. Data Model • A sparse, distributed persistent multi-dimensional sorted map (row, column, timestamp) -> cell contents

  7. DataModel • Rows • Arbitrary string • Access to data in a row is atomic • Ordered lexicographically

  8. Data Model • Column • Tow-level name structure: • family: qualifier • Column Family is the unit of access control

  9. Data Model • Timestamps • Store different versions of data in a cell • Lookup options • Return most recent K values • Return all values

  10. Data Model • The row range for a table is dynamically partitioned • Each row range is called a tablet • Tablet is the unit for distribution and load balancing

  11. APIs • Metadata operations • Create/delete tables, column families, change metadata • Writes • Set(): write cells in a row • DeleteCells(): delete cells in a row • DeleteRow(): delete all cells in a row • Reads • Scanner: read arbitrary cells in a bigtable • Each row read is atomic • Can restrict returned rows to a particular range • Can ask for just data from 1 row, all rows, etc. • Can ask for all columns, just certain column families, or specific columns

  12. APIs

  13. Building Blocks • Google File System (GFS) • stores persistent data (SSTable file format) • Scheduler • schedules jobs onto machines • Chubby • Lock service: distributed lock manager • master election, location bootstrapping • MapReduce (optional) • Data processing • Read/write Bigtable data

  14. Chubby • {lock/file/name} service • Coarse-grained locks • Each clients has a session with Chubby. • The session expires if it is unable to renew its session lease within the lease expiration time. • 5 replicas, need a majority vote to be active • Also an OSDI ’06 Paper

  15. Implementation • The Bigtable implementation has three major components: • A library that is linked into every client • One master server • Many tablet servers

  16. Tablet Location Management

  17. Refinements • Locality groups: • Clients can group multiple column families together into a locality group. • Compression: • Uses Bentley and McIlroy's scheme and fast compression algorithm. • Caching for read performance: • Uses Scan Cache and Block Cache. • Bloom filters: • Reduce the number of accesses.

  18. Performance Evaluation

  19. Real Applications • Google Analytics • http://analytics.google.com • Google Earth • http://earth.google.com • Personalized search • www.google.com/psearch

  20. Conclusions • Users like… • the performance and high availability provided by the Bigtable implementation • that they can scale the capacity of their clusters by simply adding more machines to the system as their resource demands change over time • There are significant advantages to building a custom storage solution • Challenges… • User adoption and acceptance of a new interface • Implementation issues

More Related