Multimedia Databases

1. Multimedia Databases Text I

2. Outline • Spatial Databases • Temporal Databases • Spatio-temporal Databases • Multimedia Databases • Text databases • Image and video databases • Time Series databases • Data Mining

3. Text - Detailed outline • Text databases • problem • full text scanning • inversion • signature files • clustering • information filtering and LSI

4. Problem - Motivation • Given a database of documents, find documents containing “data”, “retrieval” • Applications:

5. Problem - Motivation • …find documents containing “data”, “retrieval” • Applications: • Web • law + patent offices • digital libraries • information filtering

6. Problem - Motivation • Types of queries: • boolean (‘data’ AND ‘retrieval’ AND NOT ...) • additional features (‘data’ ADJACENT ‘retrieval’) • keyword queries (‘data’, ‘retrieval’) • How to search a large collection of documents?

7. Full-text scanning • Build a FSA; scan a c t

8. Full-text scanning • for single term: • (naive: O(N*M)) ABRACADABRA text CAB pattern

9. Full-text scanning • for single term: • (naive: O(N*M)) • Knuth Morris and Pratt (‘77) • build a small FSA; visit every text letter once only, by carefully shifting more than one step ABRACADABRA text CAB pattern

10. Full-text scanning ABRACADABRA text CAB pattern CAB ... CAB CAB

11. Full-text scanning • for single term: • (naive: O(N*M)) • Knuth Morris and Pratt (‘77) • Boyer and Moore (‘77) • preprocess pattern; start from right to left & skip! ABRACADABRA text CAB pattern

12. Full-text scanning ABRACADABRA text CAB pattern CAB CAB CAB

13. Full-text scanning ABRACADABRA text OMINOUS pattern OMINOUS Boyer+Moore: fastest, in practice Sunday (‘90): some improvements

14. Full-text scanning • For multiple terms (w/o “don’t care” characters): Aho+Corasic (‘75) • again, build a simplified FSA in O(M) time • Probabilistic algorithms: ‘fingerprints’ (Karp + Rabin ‘87) • approximate match: ‘agrep’ [Wu+Manber, Baeza-Yates+, ‘92]

15. Full-text scanning • Approximate matching - string editing distance: d(‘survey’, ‘surgery’) = 2 = min # of insertions, deletions, substitutions to transform the first string into the second SURVEY SURGERY

16. Full-text scanning • string editing distance - how to compute? • A: dynamic programming • Idea cost( i, j ) = cost to match prefix of length i of first string s with prefix of length j of second string t

17. Full-text scanning if s[i] = t[j] then cost( i, j ) = cost(i-1, j-1) else cost(i, j ) = min ( 1 + cost(i, j-1) // deletion 1 + cost(i-1, j-1) // substitution 1 + cost(i-1, j) // insertion )

18. Full-text scanning • Complexity: O(M*N) • Conclusions: Full text scanning needs no space overhead, but is slow for large datasets

19. Text - Detailed outline • text • problem • full text scanning • inversion • signature files • clustering • information filtering and LSI

20. Text – Inverted Files

21. Text – Inverted Files Q: space overhead? A: mainly, the postings lists

22. how to organize dictionary? stemming – Y/N? Keep only the root of each word ex. inverted, inversion  invert insertions? Text – Inverted Files

23. how to organize dictionary? B-tree, hashing, TRIEs, PATRICIA trees, ... stemming – Y/N? insertions? Text – Inverted Files

24. Text – Inverted Files • Other topics: • Parallelism [Tomasic+,93] • Insertions [Tomasic+94], [Brown+] • ‘zipf’ distributions • Approximate searching (‘glimpse’ [Wu+])

25. Text – Inverted Files • postings list – more Zipf distr.: eg., rank-frequency plot of ‘Bible’ log(freq) freq ~ 1/rank / ln(1.78V) log(rank)

26. Text – Inverted Files • postings lists • Cutting+Pedersen • (keep first 4 in B-tree leaves) • how to allocate space: [Faloutsos+92] • geometric progression • compression (Elias codes) [Zobel+] – down to 2% overhead!

27. Text – Inverted Files • Conclusions: needs space overhead (2%-300%), but it is the fastest

28. Text - Detailed outline • text • problem • full text scanning • inversion • signature files • clustering • information filtering and LSI

29. Signature files • idea: ‘quick & dirty’ filter

30. Signature files • idea: ‘quick & dirty’ filter • then, do seq. scan on sign. file and discard ‘false alarms’ • Adv.: easy insertions; faster than seq. scan • Disadv.: O(N) search (with small constant) • Q: how to extract signatures?

31. Signature files • A: superimposed coding!! [Mooers49], ... m (=4 bits/word) F (=12 bits sign. size)

32. Signature files • A: superimposed coding!! [Mooers49], ... data actual match

33. Signature files • A: superimposed coding!! [Mooers49], ... retrieval actual dismissal

34. Signature files • A: superimposed coding!! [Mooers49], ... nucleotic false alarm (‘false drop’)

35. Signature files • A: superimposed coding!! [Mooers49], ... ‘YES’ is ‘MAYBE’ ‘NO’ is ‘NO’

36. Signature files • Q1: How to choose F and m ? • Q2: Why is it called ‘false drop’? • Q3: other apps of signature files?

37. Signature files • Q1: How to choose F and m ? m (=4 bits/word) F (=12 bits sign. size)

38. Signature files • Q1: How to choose F and m ? • A: so that doc. signature is 50% full m (=4 bits/word) F (=12 bits sign. size)

39. Signature files • Q1: How to choose F and m ? • Q2: Why is it called ‘false drop’? • Q3: other apps of signature files?

40. Signature files • Q2: Why is it called ‘false drop’? • Old, but fascinating story [1949] • how to find qualifying books (by title word, and/or author, and/or keyword) • in O(1) time? without computers..

41. ... ... Signature files • Solution: edge-notched cards 1 2 40 • each title word is mapped to m numbers(how?) • and the corresponding holes are cut out:

42. ... ... Signature files • Solution: edge-notched cards 1 2 40 data ‘data’ -> #1, #39

43. ... ... Signature files • Search, e.g., for ‘data’: activate needle #1, #39, and shake the stack of cards! 1 2 40 data ‘data’ -> #1, #39

44. Signature files • Q3: other apps of signature files? • A: anything that has to do with ‘membership testing’: does ‘data’ belong to the set of words of the document?

45. Signature files • Another name: Bloom Filters • UNIX’s early ‘spell’ system [McIlroy] • Bloom-joins in System R* [Mackert+] and ‘active disks’ [Riedel99] • differential files [Severance+Lohman]

46. Signature files • easy insertions; slower than inversion • brilliant idea of ‘quick and dirty’ filter: quickly discard the vast majority of non-qualifying elements, and focus on the rest.

47. Aho, A. V. and M. J. Corasick (June 1975). "Fast Pattern Matching: An Aid to Bibliographic Search." CACM 18(6): 333-340. Boyer, R. S. and J. S. Moore (Oct. 1977). "A Fast String Searching Algorithm." CACM 20(10): 762-772. Brown, E. W., J. P. Callan, et al. (March 1994). Supporting Full-Text Information Retrieval with a Persistent Object Store. Proc. of EDBT conference, Cambridge, U.K., Springer Verlag. References

48. Faloutsos, C. and H. V. Jagadish (Aug. 23-27, 1992). On B-tree Indices for Skewed Distributions. 18th VLDB Conference, Vancouver, British Columbia. Karp, R. M. and M. O. Rabin (March 1987). "Efficient Randomized Pattern-Matching Algorithms." IBM Journal of Research and Development 31(2): 249-260. Knuth, D. E., J. H. Morris, et al. (June 1977). "Fast Pattern Matching in Strings." SIAM J. Comput 6(2): 323-350. References - cont’d

49. Mackert, L. M. and G. M. Lohman (August 1986). R* Optimizer Validation and Performance Evaluation for Distributed Queries. Proc. of 12th Int. Conf. on Very Large Data Bases (VLDB), Kyoto, Japan. Manber, U. and S. Wu (1994). GLIMPSE: A Tool to Search Through Entire File Systems. Proc. of USENIX Techn. Conf. McIlroy, M. D. (Jan. 1982). "Development of a Spelling List." IEEE Trans. on Communications COM-30(1): 91-99. References - cont’d

50. Mooers, C. (1949). Application of Random Codes to the Gathering of Statistical Information Bulletin 31. Cambridge, Mass, Zator Co. Pedersen, D. C. a. J. (1990). Optimizations for dynamic inverted index maintenance. ACM SIGIR. Riedel, E. (1999). Active Disks: Remote Execution for Network Attached Storage. ECE, CMU. Pittsburgh, PA. References - cont’d