1 / 97

Data Mining using Fractals and Power laws

Data Mining using Fractals and Power laws. Christos Faloutsos Carnegie Mellon University. THANK YOU!. Prof. N. Bourbakis. Overview. Goals/ motivation: find patterns in large datasets: (A) Sensor data (B) network/graph data Solutions: self-similarity and power laws Discussion.

jon
Télécharger la présentation

Data Mining using Fractals and Power laws

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Mining using Fractals and Power laws Christos Faloutsos Carnegie Mellon University C. Faloutsos

  2. THANK YOU! • Prof. N. Bourbakis C. Faloutsos

  3. Overview • Goals/ motivation: find patterns in large datasets: • (A) Sensor data • (B) network/graph data • Solutions: self-similarity and power laws • Discussion C. Faloutsos

  4. Applications of sensors/streams • ‘Smart house’: monitoring temperature, humidity etc • Financial, sales, economic series C. Faloutsos

  5. Motivation - Applications • Medical: ECGs +; blood pressure etc monitoring • Scientific data: seismological; astronomical; environment / anti-pollution; meteorological C. Faloutsos

  6. # cars Automobile traffic 2000 1800 1600 1400 1200 1000 800 600 400 200 0 time Motivation - Applications (cont’d) • civil/automobile infrastructure • bridge vibrations [Oppenheim+02] • road conditions / traffic monitoring C. Faloutsos

  7. Motivation - Applications (cont’d) • Computer systems • web servers (buffering, prefetching) • network traffic monitoring • ... http://repository.cs.vt.edu/lbl-conn-7.tar.Z C. Faloutsos

  8. Web traffic • [Crovella Bestavros, SIGMETRICS’96] C. Faloutsos

  9. survivable,self-managing storageinfrastructure a storage brick(0.5–5 TB) ~1 PB . . . . . . Self-* Storage (Ganger+) • “self-*” = self-managing, self-tuning, self-healing, … • Goal: 1 petabyte (PB) for CMU researchers • www.pdl.cmu.edu/SelfStar C. Faloutsos

  10. Problem definition • Given: one or more sequences x1 , x2 , … , xt , …; (y1, y2, … , yt, …) • Find • patterns; clusters; outliers; forecasts; C. Faloutsos

  11. Problem #1 # bytes • Find patterns, in large datasets time C. Faloutsos

  12. Problem #1 # bytes • Find patterns, in large datasets time Poisson indep., ident. distr C. Faloutsos

  13. Problem #1 # bytes • Find patterns, in large datasets time Poisson indep., ident. distr C. Faloutsos

  14. Problem #1 # bytes • Find patterns, in large datasets time Poisson indep., ident. distr Q: Then, how to generate such bursty traffic? C. Faloutsos

  15. Overview • Goals/ motivation: find patterns in large datasets: • (A) Sensor data • (B) network/graph data • Solutions: self-similarity and power laws • Discussion C. Faloutsos

  16. Problem #2 - network and graph mining • How does the Internet look like? • How does the web look like? • What constitutes a ‘normal’ social network? • What is the ‘network value’ of a customer? • which gene/species affects the others the most? C. Faloutsos

  17. Network and graph mining Food Web [Martinez ’91] Protein Interactions [genomebiology.com] Friendship Network [Moody ’01] Graphs are everywhere! C. Faloutsos

  18. Problem#2 Given a graph: • which node to market-to / defend / immunize first? • Are there un-natural sub-graphs? (eg., criminals’ rings)? [from Lumeta: ISPs 6/1999] C. Faloutsos

  19. Solutions • New tools: power laws, self-similarity and ‘fractals’ work, where traditional assumptions fail • Let’s see the details: C. Faloutsos

  20. Overview • Goals/ motivation: find patterns in large datasets: • (A) Sensor data • (B) network/graph data • Solutions: self-similarity and power laws • Discussion C. Faloutsos

  21. What is a fractal? = self-similar point set, e.g., Sierpinski triangle: zero area: (3/4)^inf infinite length! (4/3)^inf ... Q: What is its dimensionality?? C. Faloutsos

  22. What is a fractal? = self-similar point set, e.g., Sierpinski triangle: zero area: (3/4)^inf infinite length! (4/3)^inf ... Q: What is its dimensionality?? A: log3 / log2 = 1.58 (!?!) C. Faloutsos

  23. Q: fractal dimension of a line? Q: fd of a plane? Intrinsic (‘fractal’) dimension C. Faloutsos

  24. Q: fractal dimension of a line? A: nn ( <= r ) ~ r^1 (‘power law’: y=x^a) Q: fd of a plane? A: nn ( <= r ) ~ r^2 fd== slope of (log(nn) vs.. log(r) ) Intrinsic (‘fractal’) dimension C. Faloutsos

  25. log(#pairs within <=r ) 1.58 log( r ) Sierpinsky triangle == ‘correlation integral’ = CDF of pairwise distances C. Faloutsos

  26. log(#pairs within <=r ) 1.58 log( r ) Observations: Fractals <-> power laws Closely related: • fractals <=> • self-similarity <=> • scale-free <=> • power laws ( y= xa ; F=K r-2) • (vs y=e-ax or y=xa+b) C. Faloutsos

  27. Outline • Problems • Self-similarity and power laws • Solutions to posed problems • Discussion C. Faloutsos

  28. #bytes time Solution #1: traffic • disk traces: self-similar: (also: [Leland+94]) • How to generate such traffic? C. Faloutsos

  29. 20% 80% Solution #1: traffic • disk traces (80-20 ‘law’) – ‘multifractals’ #bytes time C. Faloutsos

  30. 80-20 / multifractals 20 80 C. Faloutsos

  31. 80-20 / multifractals 20 80 • p ; (1-p) in general • yes, there are dependencies C. Faloutsos

  32. More on 80/20: PQRS • Part of ‘self-* storage’ project time cylinder# C. Faloutsos

  33. p q r s More on 80/20: PQRS • Part of ‘self-* storage’ project q r s C. Faloutsos

  34. Overview • Goals/ motivation: find patterns in large datasets: • (A) Sensor data • (B) network/graph data • Solutions: self-similarity and power laws • sensor/traffic data • network/graph data • Discussion C. Faloutsos

  35. Problem #2 - topology How does the Internet look like? Any rules? C. Faloutsos

  36. Patterns? • avg degree is, say 3.3 • pick a node at random – guess its degree, exactly (-> “mode”) count ? avg: 3.3 degree C. Faloutsos

  37. Patterns? • avg degree is, say 3.3 • pick a node at random – guess its degree, exactly (-> “mode”) • A: 1!! count avg: 3.3 degree C. Faloutsos

  38. Patterns? • avg degree is, say 3.3 • pick a node at random - what is the degree you expect it to have? • A: 1!! • A’: very skewed distr. • Corollary: the mean is meaningless! • (and std -> infinity (!)) count avg: 3.3 degree C. Faloutsos

  39. -0.82 att.com log(degree) ibm.com log(rank) Solution#2: Rank exponent R • A1: Power law in the degree distribution [SIGCOMM99] internet domains C. Faloutsos

  40. Solution#2’: Eigen Exponent E Eigenvalue • A2: power law in the eigenvalues of the adjacency matrix Exponent = slope E = -0.48 May 2001 Rank of decreasing eigenvalue C. Faloutsos

  41. Power laws - discussion • do they hold, over time? • do they hold on other graphs/domains? C. Faloutsos

  42. Power laws - discussion • do they hold, over time? • Yes! for multiple years [Siganos+] • do they hold on other graphs/domains? • Yes! • web sites and links [Tomkins+], [Barabasi+] • peer-to-peer graphs (gnutella-style) • who-trusts-whom (epinions.com) C. Faloutsos

  43. att.com -0.82 log(degree) ibm.com log(rank) Time Evolution: rank R Domain level • The rank exponent has not changed! [Siganos+] C. Faloutsos

  44. The Peer-to-Peer Topology count [Jovanovic+] • Number of immediate peers (= degree), follows a power-law degree C. Faloutsos

  45. epinions.com count • who-trusts-whom [Richardson + Domingos, KDD 2001] (out) degree C. Faloutsos

  46. Why care about these patterns? • better graph generators [BRITE, INET] • for simulations • extrapolations • ‘abnormal’ graph and subgraph detection C. Faloutsos

  47. Outline • problems • Fractals • Solutions • Discussion • what else can they solve? • how frequent are fractals? C. Faloutsos

  48. What else can they solve? • separability [KDD’02] • forecasting [CIKM’02] • dimensionality reduction [SBBD’00] • non-linear axis scaling [KDD’02] • disk trace modeling [PEVA’02] • selectivity of spatial/multimedia queries [PODS’94, VLDB’95, ICDE’00] • ... C. Faloutsos

  49. Full Content Indexing, Search and Retrieval from Digital Video Archives • Query (6TB of data) • Search results • (ranked) Storyboard Collage with maps, common phrases, named entities and dynamic query sliders www.informedia.cs.cmu.edu C. Faloutsos

  50. What else can they solve? • separability [KDD’02] • forecasting [CIKM’02] • dimensionality reduction [SBBD’00] • non-linear axis scaling [KDD’02] • disk trace modeling [PEVA’02] • selectivity of spatial/multimedia queries [PODS’94, VLDB’95, ICDE’00] • ... C. Faloutsos

More Related