CS 525 Advanced Distributed Systems Spring 2014

1. CS 525 Advanced Distributed Systems Spring 2014 Indranil Gupta (Indy) Feb 11, 2014 Key-value/NoSQL Stores Lecture 7 • Based mostly on • Cassandra NoSQL presentation • Cassandra 1.0 documentation at datastax.com • Cassandra Apache project wiki • HBase  2014, I. Gupta

2. Cassandra • Originally designed at Facebook • Open-sourced • Some of its myriad users: • With this many users, one would think • Its design is very complex • Let’s find out!

3. Why Key-value Store? (Business) Key -> Value (twitter.com) tweet id -> information about tweet (amazon.com) item number -> information about it (kayak.com) Flight number -> information about flight, e.g., availability (yourbank.com) Account number -> information about it Search is usually built on top of a key-value store

4. Isn’t that just a database? • Yes, sort of • Relational Databases (RDBMSs) have been around for ages • MySQL is the most popular among them • Data stored in tables • Schema-based, i.e., structured tables • Queried using SQL (Structured Query Language) SQL queries: SELECT user_id from users WHERE username = “jbellis” Example’s Source

5. Mismatch with today’s workloads • Data: Large and unstructured • Lots of random reads and writes • Foreign keys rarely needed • Need • Speed • Avoid Single point of Failure (SPoF) • Low TCO (Total cost of operation) and fewer sysadmins • Incremental Scalability • Scale out, not up: use more machines that are off the shelf (COTS), not more powerful machines

6. CAP Theorem • Proposed by Eric Brewer (Berkeley) • Subsequently proved by Gilbert and Lynch • In a distributed system you can satisfy at most 2 out of the 3 guarantees • Consistency: all nodes see same data at any time, or reads return latest written value • Availability: the system allows operations all the time • Partition-tolerance: the system continues to work in spite of network partitions • Cassandra • Eventual (weak) consistency, Availability, Partition-tolerance • Traditional RDBMSs • Strong consistency over availability under a partition

7. CAP Tradeoff Consistency • Starting point for NoSQL Revolution • Conjectured by Eric Brewer in 2000, proved by Gilbert and Lynch in 2002 • A distributed storage system can achieve at most two of C, A, and P. • When partition-tolerance is important, you have to choose between consistency and availability HBase, HyperTable, BigTable, Spanner RDBMSs Partition-tolerance Availability Cassandra, RIAK, Dynamo, Voldemort

8. Eventual Consistency • If all writers stop (to a key), then all its values (replicas) will converge eventually. • If writes continue, then system always tries to keep converging. • Moving “wave” of updated values lagging behind the latest values sent by clients, but always trying to catch up. • May return stale values to clients (e.g., if many back-to-back writes). • But works well when there a few periods of low writes. System converges quickly.

9. Cassandra – Data Model • Column Families: • Like SQL tables • but may be unstructured (client-specified) • Can have index tables • “Column-oriented databases”/ “NoSQL” • Columns stored together, rather than rows • No schemas • Some columns missing from some entries • NoSQL = “Not Only SQL” • Supports get(key) and put(key, value) operations • Often write-heavy workloads

10. Let’s go Inside Cassandra: Key -> Server Mapping • How do you decide which server(s) a key-value resides on?

11. (Remember this?) Say m=7 0 N16 N112 Primary replica for key K13 N96 N32 Read/write K13 N45 N80 Coordinator (typically one per DC) Backup replicas for key K13 Cassandra uses a Ring-based DHT but without routing Key->server mapping is the “Partitioning Function”

12. Writes • Need to be lock-free and fast (no reads or disk seeks) • Client sends write to one front-end node in Cassandra cluster • Front-end = Coordinator, assigned per key • Which (via Partitioning function) sends it to all replica nodes responsible for key • Always writable: Hinted Handoff mechanism • If any replica is down, the coordinator writes to all other replicas, and keeps the write locally until down replica comes back up. • When all replicas are down, the Coordinator (front end) buffers writes (for up to a few hours). • Provides Atomicity for a given key (i.e., within a ColumnFamily) • One ring per datacenter • Per-DC coordinator elected to coordinate with other DCs • Election done via Zookeeper, which runs a Paxos variant

13. Writes at a replica node On receiving a write • 1. log it in disk commit log • 2. Make changes to appropriate memtables • In-memory representation of multiple key-value pairs • Later, when memtable is full or old, flush to disk • Data File: An SSTable (Sorted String Table) – list of key value pairs, sorted by key • Index file: An SSTable of (key, position in data sstable) pairs • And a Bloom filter (for efficient search) – next slide • Compaction: Data udpates accumulate over time and SStables and logs need to be compacted • Merge SSTables, e.g., by merging updates for a key • Run periodically and locally at each server • Reads need to touch log and multiple SSTables • A row may be split over multiple SSTables • Reads may be slower than writes

14. Bloom Filter • Compact way of representing a set of items • Checking for existence in set is cheap • Some probability of false positives: an item not in set may check true as being in set • Never false negatives Large Bit Map 0 • On insert, set all hashed bits. • On check-if-present, • return true if all hashed bits set. • False positives • False positive rate low • k=4 hash functions • 100 items • 3200 bits • FP rate = 0.02% 1 2 3 Hash1 Key-K Hash2 . . 69 Hashk 111 127

15. Deletes and Reads • Delete: don’t delete item right away • Add a tombstone to the log • Compaction will eventually remove tombstone and delete item • Read: Similar to writes, except • Coordinator can contacts a number of replicas (e.g., in same rack) specified by consistency level • Forwards read to replicas that have responded quickest in past • Returns latest timestamp value • Coordinator also fetches value from multiple replicas • check consistency in the background, initiating a read-repair if any two values are different • Brings all replicas up to date • A row may be split across multiple SSTables => reads need to touch multiple SSTables => reads slower than writes (but still fast)

16. Cassandra uses Quorums • Quorum = way of selecting sets so that any pair of sets intersect • E.g., any arbitrary set with at least Q=N/2 +1 nodes • Where N = total number of replicas for this key • Reads • Wait for R replicas (R specified by clients) • In the background, check for consistency of remaining N-R replicas, and initiate read repair if needed • Writes come in two default flavors • Block until quorum is reached • Async: Write to any node • R = read replica count, W = write replica count • If W+R > N and W > N/2, you have consistency, i.e., each read returns the latest written value • Reasonable: (W=1, R=N) or (W=N, R=1) or (W=Q, R=Q)

17. Cassandra Consistency Levels • In reality, a client can choose one of these levels for a read/write operation: • ANY: any node (may not be replica) • ONE: at least one replica • Similarly, TWO, THREE • QUORUM: quorum across all replicas in all datacenters • LOCAL_QUORUM: in coordinator’s DC • EACH_QUORUM: quorum in every DC • ALL: all replicas all DCs • For Write, you also have • SERIAL: claims to implement linearizability for transactions

18. Cluster Membership – Gossip-Style Cassandra uses gossip-based cluster membership 2 1 Address Time (local) Heartbeat Counter • Protocol: • Nodes periodically gossip their membership list • On receipt, the local membership list is updated, as shown • If any heartbeat older than Tfail, node is marked as failed 4 3 Current time : 70 at node 2 (asynchronous clocks) Fig and animation by: Dongyun Jin and Thuy Ngyuen

19. Gossip-Style Failure Detection • If the heartbeat has not increased for more than Tfail seconds (according to local time), the member is considered failed • But don’t delete it right away • Wait an additional Tfail seconds, then delete the member from the list • Why?

20. Gossip-Style Failure Detection • What if an entry pointing to a failed process is deleted right after Tfail (= 24) seconds? • Fix: remember for another Tfail • Ignore gossips for failed members • Don’t include failed members in gossip messages 2 1 Current time : 75 at process 2 4 3

21. Cluster Membership, contd. • Suspicion mechanisms to adaptively set the timeout • Accrual detector: Failure Detector outputs a value (PHI) representing suspicion • Apps set an appropriate threshold • PHI = 5 => 10-15 sec detection time • PHI calculation for a member • Inter-arrival times for gossip messages • PHI(t) = - log(CDF or Probability(t_now – t_last))/log 10 • PHI basically determines the detection timeout, but takes into account historical inter-arrival time variations for gossiped heartbeats

22. Data Placement Strategies • Replication Strategy: two options: • SimpleStrategy • NetworkTopologyStrategy • SimpleStrategy: uses the Partitioner • RandomPartitioner: Chord-like hash partitioning • ByteOrderedPartitioner: Assigns ranges of keys to servers. • Easier for range queries (e.g., Get me all twitter users starting with [a-b]) • NetworkTopologyStrategy: for multi-DC deployments • Two replicas per DC: allows a consistency level of ONE • Three replicas per DC: allows a consistency level of LOCAL_QUORUM • Per DC • First replica placed according to Partitioner • Then go clockwise around ring until you hit different rack

23. Snitches • Maps: IPs to racks and DCs. Configured in cassandra.yaml config file • Some options: • SimpleSnitch: Unaware of Topology (Rack-unaware) • RackInferring: Assumes topology of network by octet of server’s IP address • 101.201.301.401 = x.<DC octet>.<rack octet>.<node octet> • PropertyFileSnitch: uses a config file • EC2Snitch: uses EC2. • EC2 Region = DC • Availability zone = rack • Other snitch options available

24. Vs. SQL • MySQL is one of the most popular (and has been for a while) • On > 50 GB data • MySQL • Writes 300 ms avg • Reads 350 ms avg • Cassandra • Writes 0.12 ms avg • Reads 15 ms avg

25. Cassandra Summary • While RDBMS provide ACID (Atomicity Consistency Isolation Durability) • Cassandra provides BASE • Basically Available Soft-state Eventual Consistency • Prefers Availability over consistency • Other NoSQL products • MongoDB, Riak (look them up!) • Next: HBase • Prefers (strong) Consistency over Availability

26. HBase • Google’s BigTable was first “blob-based” storage system • Yahoo! Open-sourced it -> HBase • Major Apache project today • Facebook uses HBase internally • API • Get/Put(row) • Scan(row range, filter) – range queries • MultiPut

27. HBase Architecture Small group of servers running Zab, a consensus protocol (Paxos-like) HDFS Source: http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html

28. HBase Storage hierarchy • HBase Table • Split it into multiple regions: replicated across servers • One Store per combination of ColumnFamily (subset of columns with similar query patterns) + region • Memstore for each Store: in-memory updates to Store; flushed to disk when full • StoreFiles for each store for each region: where the data lives - Blocks • HFile • SSTable from Google’s BigTable

29. HFile (For a census table example) Ethnicity SSN:000-01-2345 Demographic Source: http://blog.cloudera.com/blog/2012/06/hbase-io-hfile-input-output/

30. Strong Consistency: HBase Write-Ahead Log Write to HLog before writing to MemStore Thus can recover from failure Source: http://www.larsgeorge.com/2010/01/hbase-architecture-101-write-ahead-log.html

31. Log Replay • After recovery from failure, or upon bootup (HRegionServer/HMaster) • Replay any stale logs (use timestamps to find out where the database is w.r.t. the logs) • Replay: add edits to the MemStore • Keeps one HLog per HRegionServer rather than per region • Avoids many concurrent writes, which on the local file system may involve many disk seeks

32. Cross-data center replication HLog Zookeeper is actually a file system for control information 1. /hbase/replication/state 2. /hbase/replication/peers /<peer cluster number> 3. /hbase/replication/rs/<hlog>

33. Wrap Up • Key-value stores and NoSQL trade off between Availability and Consistency, while providing partition-tolerance • Presentations and reviews start this Thursday • Presenters: please email me slides at least 24 hours before your presentation time • Everyone: please read instructions on website • If you have not signed up for a presentation slot, you’ll need to do all reviews. • Project meetings starting soon.