Hash-Based Indexes

1. Hash-Based Indexes Dynamic Hash indexes Extendible, Linear indexes

2. Introduction • Hash-based indexes are best for equalityselections • no traversal; direct computation of where k* should be • cannot support range searches. • Static and dynamic hashing techniques exist;

3. Static Hashing • # primary pages fixed, allocated sequentially, never de-allocated; overflow pages if needed. • h(k) mod M = bucket to which data entry withkey k (i.e., k*) belongs. (M = # of buckets) 0 h(key) mod M 1 key h M-1 Primary bucket pages Overflow pages

4. Dynamic Hashing • Good for database that grows and shrinks in size • Allows the hash function to be modified dynamically • Extendable hashing – one form of dynamic hashing • Hash function generates values over a large range — typically b-bit integers, with b = 32. • At any time use only a prefix of the hash function to index into a table of bucket addresses. • Let the length of the prefix be i bits, 0  i  32. • Bucket address table size = 2i.Initially i = 0 • Value of i grows and shrinks as the size of the database grows and shrinks. • Multiple entries in the bucket address table may point to a bucket • Thus, actual number of buckets is < 2i • The number of buckets also changes dynamically due to coalescing and splitting of buckets.

5. General Extendable Hash Structure In this structure, i2 = i3 = i, whereas i1 = i– 1

6. Extendible Hashing • Situation: Bucket (primary page) becomes full. Why not re-organize file by doubling # of buckets? • Reading and writing all pages is expensive! • Idea: Use directory of pointers to buckets, double # of buckets by doubling the directory, splitting just the bucket that overflowed! • Directory much smaller than file, so doubling it is much cheaper. Only one page of data entries is split. Nooverflowpage!

7. LOCAL DEPTH 2 Example Bucket A 16* 4* 12* 32* GLOBAL DEPTH 2 2 Bucket B 00 5* 1* 21* 13* 01 • Directory is array of size 4. • To find bucket for r, take last `global depth’ # bits of h(r); we denote r by h(r). • If h(r) = 5 = binary 101, it is in bucket pointed to by 01. 2 10 Bucket C 10* 11 2 DIRECTORY Bucket D 15* 7* 19* DATA PAGES • Insert: If bucket is full, splitit (allocate new page, re-distribute). • If necessary, double the directory. (As we will see, splitting a • bucket does not always require doubling; we can tell by • comparing global depth with local depth for the split bucket.)

8. Insert h(r)=20 (Causes Doubling) 2 LOCAL DEPTH 3 LOCAL DEPTH Bucket A 16* 32* 32* 16* GLOBAL DEPTH Bucket A GLOBAL DEPTH 2 2 2 3 Bucket B 5* 21* 13* 1* 00 1* 5* 21* 13* 000 Bucket B 01 001 2 10 2 010 Bucket C 10* 11 10* Bucket C 011 100 2 2 DIRECTORY 101 Bucket D 15* 7* 19* 15* 7* 19* Bucket D 110 111 2 3 Bucket A2 4* 12* 20* DIRECTORY 12* 20* Bucket A2 4* (`split image' of Bucket A) (`split image' of Bucket A)

9. Points to Note • 20 = binary 10100. Last 2 bits (00) tell us r belongs in A or A2. Last 3 bits needed to tell which. • Global depth of directory: Max # of bits needed to tell which bucket an entry belongs to. • Local depth of a bucket: # of bits used to determine if an entry belongs to this bucket. • When does bucket split cause directory doubling? • Before insert, local depth of bucket = global depth. Insert causes local depth to become > global depth; directory is doubled by copying it over and `fixing’ pointer to split image page. (Use of least significant bits enables efficient doubling via copying of directory!)

10. Comments on Extendible Hashing • If directory fits in memory, equality search answered with one disk access; else two. • 100MB file, 100 bytes/rec, 4K pages contains 1,000,000 records (as data entries) and 25,000 directory elements; chances are high that directory will fit in memory. • Directory grows in spurts, and, if the distribution of hash values is skewed, directory can grow large. • Multiple entries with same hash value cause problems! • Delete: If removal of data entry makes bucket empty, can be merged with `split image’. If each directory element points to same bucket as its split image, can halve directory.

11. Use of Extendable Hash Structure • Each bucket j stores a value ij • All the entries that point to the same bucket have the same values on the first ij bits. • To locate the bucket containing search-key Kj: 1. Compute h(Kj) = X 2. Use the first i high order bits of X as a displacement into bucket address table, and follow the pointer to appropriate bucket • To insert a record with search-key value Kj • follow same procedure as look-up and locate the bucket, say j. • If there is room in the bucket j insert record in the bucket. • Else the bucket must be split and insertion re-attempted (next slide.) • Overflow buckets used instead in some cases (will see shortly)

12. Insertion in Extendable Hash Structure (Cont) To split a bucket j when inserting record with search-key value Kj: • If i > ij (more than one pointer to bucket j) • allocate a new bucket z, and set ij= iz = (ij + 1) • Update the second half of the bucket address table entries originally pointing to j, to point to z • remove each record in bucket j and reinsert (in j or z) • recompute new bucket for Kjand insert record in the bucket (further splitting is required if the bucket is still full) • If i = ij(only one pointer to bucket j) • If i reaches some limit b, or too many splits have happened in this insertion, create an overflow bucket • Else • increment i and double the size of the bucket address table. • replace each entry in the table by two entries that point to the same bucket. • recompute new bucket address table entry for KjNow i > ij so use the first case above.

13. Deletion in Extendable Hash Structure • To delete a key value, • locate it in its bucket and remove it. • The bucket itself can be removed if it becomes empty (with appropriate updates to the bucket address table). • Coalescing of buckets can be done (can coalesce only with a “buddy” bucket having same value of ij and same ij –1 prefix, if it is present) • Decreasing bucket address table size is also possible • Note: decreasing bucket address table size is an expensive operation and should be done only if number of buckets becomes much smaller than the size of the table

15. Use of Extendable Hash Structure: Example

16. Example (Cont.) • Initial Hash structure; bucket size = 2

17. Example (Cont.) • Hash structure after insertion of “Mozart”, “Srinivasan”, and “Wu” records

18. Example (Cont.) • Hash structure after insertion of Einstein record

19. Example (Cont.) • Hash structure after insertion of Gold and El Said records

20. Example (Cont.) • Hash structure after insertion of Katz record

21. Example (Cont.) And after insertion of eleven records

22. Example (Cont.) And after insertion of Kim record in previous hash structure

23. Extendable Hashing vs. Other Schemes • Benefits of extendable hashing: • Hash performance does not degrade with growth of file • Minimal space overhead • Disadvantages of extendable hashing • Extra level of indirection to find desired record • Bucket address table may itself become very big (larger than memory) • Cannot allocate very large contiguous areas on disk either • Solution: B+-tree structure to locate desired record in bucket address table • Changing size of bucket address table is an expensive operation • Linear hashingis an alternative mechanism • Allows incremental growth of its directory (equivalent to bucket address table) • At the cost of more bucket overflows

24. Linear Hashing • This is another dynamic hashing scheme, an alternative to Extendible Hashing. • LH handles the problem of long overflow chains without using a directory, and handles duplicates. • Idea: Use a family of hash functions h0, h1, h2, ... • hi(key) = h(key) mod(2iN); N = initial # buckets • h is some hash function (range is not 0 to N-1) • If N = 2d0, for some d0, hi consists of applying h and looking at the last di bits, where di = d0 + i. • hi+1 doubles the range of hi (similar to directory doubling)

25. Linear Hashing (Contd.) • Directory avoided in LH by using overflow pages, and choosing bucket to split round-robin. • Splitting proceeds in `rounds’. Round ends when all NRinitial (for round R) buckets are split. Buckets 0 to Next-1 have been split; Next to NR yet to be split. • Current round number is Level. • Search:To find bucket for data entry r, findhLevel(r): • If hLevel(r) in range `Next to NR’, r belongs here. • Else, r could belong to bucket hLevel(r) or bucket hLevel(r) + NR; must apply hLevel+1(r) to find out.

26. Overview of LH File • In the middle of a round. Buckets split in this round: Bucket to be split If ( h search key value ) Level Next is in this range, must use h ( search key value ) Level+1 Buckets that existed at the to decide if entry is in beginning of this round: `split image' bucket. this is the range of h Level `split image' buckets: created (through splitting of other buckets) in this round

27. Linear Hashing (Contd.) • Insert: Find bucket by applying hLevel / hLevel+1: • If bucket to insert into is full: • Add overflow page and insert data entry. • (Maybe) Split Next bucket and increment Next. • Can choose any criterion to `trigger’ split. • Since buckets are split round-robin, long overflow chains don’t develop! • Doubling of directory in Extendible Hashing is similar; switching of hash functions is implicit in how the # of bits examined is increased.

28. Example of Linear Hashing Level=0, N=4 • starts with 4 buckets • all buckets to be split in a round-robin fashion, starting from the first one PRIMARY h h PAGES 0 1 Next=0 32* 44* 36* 000 00 Data entry r 9* 5* 25* with h(r)=5 001 01 30* 10* 14* 18* Primary 10 010 bucket page 31* 35* 7* 11* 011 11 (This info is for illustration only!) (The actual contents of the linear hashed file)

29. Example – Inserting 43* • h0 (43) = 11 => overflow • overflow page exists! • splitting occurs but to the Next bucket Level=0 h OVERFLOW h PRIMARY 0 1 PAGES PAGES 32* 000 00 Next=1 9* 5* 25* 001 01 30* 10* 14* 18* 10 010 31* 35* 7* 11* 43* 011 11 100 44* 36* 00

30. Example: End of a Round (Inserting 37*,29*, 22*,66*,34*,50*) Level=1 PRIMARY OVERFLOW h h PAGES 0 1 PAGES Next=0 Level=0 00 000 32* PRIMARY OVERFLOW PAGES h PAGES h 1 0 001 01 9* 25* 32* 000 00 10 010 50* 10* 18* 66* 34* 9* 25* 001 01 011 11 35* 11* 43* 66* 10 18* 10* 34* 010 Next=3 100 00 44* 36* 43* 11* 7* 31* 35* 011 11 101 11 5* 29* 37* 44* 36* 100 00 14* 22* 30* 110 10 5* 37* 29* 101 01 14* 30* 22* 31* 7* 11 111 110 10

31. Linear Hashing - Searching • Search:To find bucket for data entry r, findhLevel(r): • If hLevel(r) in range `Next to NR-1’, r belongs here. • Else, r could belong to bucket hLevel(r) or bucket hLevel(r) + NR ; must apply hLevel+1(r) to find out.

32. LH – Deletion • Inverse of insertion. • If last bkt is empty, remove it and decrement Next. • More generally, can combine last bkt with its split image even if non-empty. Criterion may be based on bkt occupancy level.

33. LH – Deletion (example) After deleting 14*, 22* Level=0 Level=0 PRIMARY OVERFLOW h h PAGES 1 0 h PAGES h 1 0 32* 000 00 32* 000 00 Delete 30* 9* 25* 001 01 9* 25* 001 01 Next=2 66* 10 18* 10* 34* 010 66* 10 18* 10* 34* 010 Next=3 Next=3 43* 11* 7* 31* 35* 011 11 43* 11* 7* 31* 35* 011 11 44* 36* 100 00 44* 36* 100 00 5* 37* 29* 101 01 5* 37* 29* 101 01 30* 110 30* 10 110 10

34. Summary • Hash-based indexes: best for equality searches, cannot support range searches. • Static Hashing can lead to long overflow chains. • Extendible Hashing avoids overflow pages by splitting a full bucket when a new data entry is to be added to it. • Directory to keep track of buckets, doubles periodically. • Can get large with skewed data; additional I/O if this does not fit in main memory.

35. Summary (Contd.) • Linear Hashing avoids directory by splitting buckets round-robin, and using overflow pages. • Overflow pages not likely to be long. • Duplicates handled easily. • Space utilization could be lower than Extendible Hashing, since splits not concentrated on `dense’ data areas. • Can tune criterion for triggering splits to trade-off slightly longer chains for better space utilization. • For hash-based indexes, a skewed data distribution is one in which the hash values of data entries are not uniformly distributed!

36. Comparison of Ordered Indexing and Hashing • Cost of periodic re-organization • Relative frequency of insertions and deletions • Is it desirable to optimize average access time at the expense of worst-case access time? • Expected type of queries: • Hashing is generally better at retrieving records having a specified value of the key. • If range queries are common, ordered indices are to be preferred • In practice: • PostgreSQL supports hash indices, but discourages use due to poor performance • Oracle supports static hash organization, but not hash indices • SQLServer supports only B+-trees

37. Bitmap Indices • Bitmap indices are a special type of index designed for efficient querying on multiple keys • Records in a relation are assumed to be numbered sequentially from, say, 0 • Given a number n it must be easy to retrieve record n • Particularly easy if records are of fixed size • Applicable on attributes that take on a relatively small number of distinct values • E.g. gender, country, state, … • E.g. income-level (income broken up into a small number of levels such as 0-9999, 10000-19999, 20000-50000, 50000- infinity) • A bitmap is simply an array of bits

38. Bitmap Indices (Cont.) • In its simplest form a bitmap index on an attribute has a bitmap for each value of the attribute • Bitmap has as many bits as records • In a bitmap for value v, the bit for a record is 1 if the record has the value v for the attribute, and is 0 otherwise

39. Bitmap Indices (Cont.) • Bitmap indices are useful for queries on multiple attributes • not particularly useful for single attribute queries • Queries are answered using bitmap operations • Intersection (and) • Union (or) • Complementation (not) • Each operation takes two bitmaps of the same size and applies the operation on corresponding bits to get the result bitmap • E.g. 100110 AND 110011 = 100010 100110 OR 110011 = 110111 NOT 100110 = 011001 • Males with income level L1: 10010 AND 10100 = 10000 • Can then retrieve required tuples. • Counting number of matching tuples is even faster

40. Bitmap Indices (Cont.) • Bitmap indices generally very small compared with relation size • E.g. if record is 100 bytes, space for a single bitmap is 1/800 of space used by relation. • If number of distinct attribute values is 8, bitmap is only 1% of relation size • Deletion needs to be handled properly • Existence bitmapto note if there is a valid record at a record location • Needed for complementation • not(A=v): (NOT bitmap-A-v) AND ExistenceBitmap • Should keep bitmaps for all values, even null value • To correctly handle SQL null semantics for NOT(A=v): • intersect above result with (NOT bitmap-A-Null)

41. Efficient Implementation of Bitmap Operations • Bitmaps are packed into words; a single word and (a basic CPU instruction) computes and of 32 or 64 bits at once • E.g. 1-million-bit maps can be and-ed with just 31,250 instruction • Counting number of 1s can be done fast by a trick: • Use each byte to index into a precomputed array of 256 elements each storing the count of 1s in the binary representation • Can use pairs of bytes to speed up further at a higher memory cost • Add up the retrieved counts • Bitmaps can be used instead of Tuple-ID lists at leaf levels of B+-trees, for values that have a large number of matching records • Worthwhile if > 1/64 of the records have that value, assuming a tuple-id is 64 bits • Above technique merges benefits of bitmap and B+-tree indices

42. Index Definition in SQL • Create an index create index <index-name> on <relation-name> (<attribute-list>) E.g.: create index b-index on branch(branch_name) • Use create unique index to indirectly specify and enforce the condition that the search key is a candidate key is a candidate key. • Not really required if SQL unique integrity constraint is supported • To drop an index drop index <index-name> • Most database systems allow specification of type of index, and clustering.