1 / 50

Data Streams

Data Streams. Definition: Data arriving continuously, usually just by insertions of new elements. The size of the stream is not known a-priori, and may be unbounded. Hot research area. Data Streams. Applications: Phone call records in AT&T Network Monitoring

hinda
Télécharger la présentation

Data Streams

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Data Streams • Definition: Data arriving continuously, usually just by insertions of new elements. The size of the stream is not known a-priori, and may be unbounded. • Hot research area

  2. Data Streams • Applications: • Phone call records in AT&T • Network Monitoring • Financial Applications (Stock Quotes) • Web Applications (Data Clicks) • Sensor Networks

  3. Continuous Queries • Mainly used in data stream environments • Defined once, and run until user terminates them • Example: Give me the names of the stocks who increased their value by at least 5% over the last hour

  4. What is the Problem (1)? • Q is a selection: then size(A) may be unbounded. Thus, we cannot guarantee we can store it.

  5. What is the Problem (2)? • Q is a self-join: If we want to provide only NEW results, then we need unlimited storage to guarantee no duplicates exist in result

  6. What is the Problem (3)? • Q contains aggregation: then tuples in A might be deleted by new observed tuples. Ex: Select A, sum(B) From Stream X What if B < 0 ? Group by A Having sum(B) > 100

  7. What is the Problem (4)? • What if we can delete tuples in the Stream? • What if Q contains a blocking operator near the top (example: aggregation)? • Online Aggregation Techniques useful

  8. Global Architecture

  9. Related Areas • Data Approximation: limits size of scratch, store • Grouping Continuous Queries submitted over the same sources • Adaptive Query Processing (data sources may be providing elements with varying rates) • Partial Results: Give partial results to the user (the query may run forever) • Data Mining: Can the algorithms be modified to use one scan of the data and still provide good results?

  10. Initial Approaches (1) • Typical Approach: Limit expressiveness of query language to limit size of Store, Scratch • Alert (1991): • Triggers on Append-Only (Active) Tables • Event-condition-action triggers • Event: Cursor on Active Table • Condition: From and Where Clause of Rule • Action: Select Clause of Rule (typically called a function) • Triggers were expressed as continuous queries • User was responsible for monitoring the size of tables

  11. Initial Approaches (2) • Tapestry (1992): • Introduced Notion of Continuous Queries • Used subset of SQL (TQL) • Query Q was converted to the Minimum Monotone Bounding Query QM(t)= Union QM(τ) , for all τ <= t • Then QM was converted to an Incremental query QI. • Problems: • Duplicate tuples were returned • Aggregation Queries were not supported • No Outer-Joins allowed

  12. Initial Approaches (3) • Chronicle Data Model (1995): • Data Streams referred as Chronicles (append-only) • Assumptions: • A new tuple is not joined with previously seen tuples. • At most a constant number of tuples from a relation R can join with Chronicle C. • Achievement: Incremental maintenance of views in time independent of the Chronicle size

  13. Materialized Views • Work on Self-Maintenance: important to limit size of Scratch. If a view can be self-maintainable, any auxiliary storage much occupy bounded space • Work on Data Expiration: important for knowing when to move elements from Scratch to Throw.

  14. Data Approximation • Area most working is being done nowadays • Problem: We cannot have O(N) space/time cost per element to solve a problem, but want solutions close to O(poly(logN)). • Sampling • Histograms • Wavelets • Sketching Techniques

  15. Sampling • Easiest one to implement, use • Reservoir Sampling: dominant algorithm • Used for any problem (but with serious limitations, especially in cases of joins) • Stratified Sampling (sampling data at different rates) • Reduce variance in data • Reduce error in Group-By Queries.

  16. Histograms • V-Optimal: • Gilbert et al. removed sorted restriction: time/space using sketches in O(poly(B,logN,1/ε) • Equi-Width: • Compute quantiles in O(1/ε logεN) space and precision of εN. • Correlated Aggregates: • AGG-D{Y : Predicate(X, AGG-I(X)) } • AGG-D : Count or Sum • AGG-I : Min, Max or Average • Reallocate histogram based on arriving tuples, and the AGG-I (if we want min, and are storing [min, min + ε] in the histogram and receive new min, throw away previous histogram.

  17. Wavelets • Used for Signal Decomposition (good if measured aggregate follows a signal) • Matias, Vitter: Incremental Maintenance of top Wavelet coefficients • Gilbert et al: Point, Range Queries with wavelets

  18. Sketching Techniques (1) • Main idea: If getting the exact value of a variable V requires O(n) time, then use approximation: • Define a random variable R with expected value equal to that of V, and small variance. • Example (self-join): • Select 4-wise independent variables ξi (i = 1, …, dom(A)) • Define Z = X2, X = Σf(i)ξ(i) , f(i): frequency of i-th value • Result is median of s2 variables Yj, where Yj is the average of s1 variables (boosting accuracy, confidence interval)

  19. Sketching Techniques (2) • Answer Complex Aggregate Queries • Frequency moments Fk, where capture statistics of data • mi: the frequency of occurrence for value i • F0: number of distinct values • F1: number of total elements • F2: Gini index (useful for self-joins) • L1, L2 norms of a vector computes similarly to F2 • Quantiles (combination of histograms, sketches)

  20. Grouping Continuous Queries • Goal: Group similar queries over data sources, to eliminate common processing needed and minimize response time and storage needed. • Niagara (joint work of Wisconsin, Oregon) • Tukwilla (Washington) • Telegraph (Berkeley)

  21. Niagara (1) • Supports thousands of queries over XML sources. • Features: • Incremental Grouping of Queries • Supports queries evaluated when data sources change (change-based) • Supports queries evaluated at specific intervals (timer-based) • Timer-based queries are harder to group because of overlapping time intervals • Change-based queries have better response times but waste more resources.

  22. Niagara (2) • Why Group Queries: • Share Computation • Test multiple “Fire” actions together • Group plans can be kept in memory more easily

  23. Niagara - Key ideas • Query Expression Signature • Query Plan (generated by Niagara parser)

  24. Group • Group signature (common signature of all queries in a plan) • Group Constant Table (signature constants of all the queries in the group, and destination buffers) • Group plan: the query plan shared by all queries in the group.

  25. Incremental Grouping • Create signature of new query. Place in lower parts of signature the most selective predicates. • Insert new query in the Group which best matches its signature bottom-up • If no match is found, create new group for this query • Store any timer information, and the data sources needed for this query

  26. Other Issues (1) • Why write output to file, and not use pipelining?: • Pipelining would fire all actions, even if new needed to be fired • Pipelining does not work in timer-based queries where results need to be buffered. • Split operator may become a bottleneck if output is consumed in widely different rates • Query plan too complex for the optimizer

  27. Other Issues (2) • Selection Operator above, or below Joins? • Below only if selections are very selective • Else, better to have one join • Range queries? • Like equality queries. Save lower, upper bound • Output in one common sorted file to eliminate duplicates.

  28. Tukwilla • Adaptive Query Processing over autonomous data sources • Periodically change query plan if output of operators is not satisfactory. • At the end perform cleanup. Some calculated results may have to be thrown away.

  29. Telegraph • Adaptive Query Engine based on Eddy Concept • Queries are over autonomous sources over the internet • Environment is unpredictable and data rates may differ significantly during query execution. Therefore the query processing SHOULD be adaptive. • Also can help produce partial results

  30. Eddy • Eddy: Routes tuples to operators for processing, gets them back and routes them again…

  31. Eddy – Knowing State of Tuples • Passes Tuples by Reference to Operators (avoids copying) • When Eddy does not have any more input tuples, it polls the sources for more input. • Tuples need to be augmented with additional information: • Ready Bits: Which operators need to be applied • Done Bits: Which operators have been applied • Queries Completed: Signals if tuple has been output or rejected by the query • Completion Mask (per query): To know when a tuple can be output for a query (completion mask & done bits = mask)

  32. Eddy – Other Details • Queries with no joins are partitioned per data source (to save space in the bits required) • Queries with Disjunctions (OR’s) are transformed into conjunctive normal form (and of or’s). • Range/exact predicates are found in Grouped filter

  33. Joins - SteMs • SteMs: Multiway-Pipelined Joins • Double- Pipelined Joins maintain a hash index on each relation. • When N relations are joined, at least n-2 inflight indices are needed for intermediate results even for left-deep trees. • Previous approach cannot change query plan without re-computing intermediate indices.

  34. SteMs - Functionality • Keeps hash-table (or other index) on one data source • Can have tuples inserted into it (passed from eddy) • Can be probed. Intermediate tuples (results of join) are returned to eddy, with the appropriate bits set • Tuples have sequence numbers. A tuple X can join only with tuples in stem M, if the indexed tuples have lower sequence numbers than X (arrived earlier).

  35. Telegraph – Routing • How to route between operators? • Route to operator with smaller queue • Route to more selective operators (ticket scheme)

  36. Partial Results - Telegraph • Idea: When tuple returns to eddy, the tuple may contribute to final result (fields may be missing because of joins not performed yet). • Present tuple anyway. The missing fields will be filled later. • Tuple is guaranteed to be in result if referential constraints exist (foreign keys). Not usual in web sources. • Might be useful to the user to present tuples that do not have matching fields (like in outer join).

  37. Partial Results - Telegraph • Results presented in tabular representation • User can: • Re-arrange columns • Drill down (add columns) or roll-up (remove columns) • Assume current area of focus is where user needs more tuples. • Weight tuples based on: • Selected columns and their order • Selected Values for some dimension • Eddy sorts tuples according to their benefit in result and schedule them accordingly

  38. Partial Results – Other methods • Online Aggregation: Present current aggregate with error bounds, and continuously refine results • Previous approaches involved changing some blocking operators to be able to get partial results • Join (use symmetric hash-join) • Nest • Average • Except

  39. Data Mining (1) • General problem: Data mining techniques usually require: • Entire dataset to be present (in memory or in disk) • Multiple passes of the data • Too much time per data element

  40. Data Mining (2) • New algorithms should require: • Small constant time per record • Use of a fixed amount of memory • Use one scan of data • Provide a useful model at all times • Produce a model that would be close to the one produced by multiple passes over the same data if the dataset was available offline. • Alter the model when generating phenomenon changes over time

  41. Decision Trees • Input: A set of examples (x, v) where x is a vector of D attributes and v is a discrete class label • Find at each node the best attribute to split. • Hoeffding bounds are useful here: • Consider a variable r with range R • N independent observations • Computed average r’ differs by true average of r by at most ε with probability 1-δ, where:

  42. Hoeffding Tree • At each node maintain counts for each attribute X, and each value Xi of X and each correct class • Let G(Xi) be the heuristic measure to choose test attributes (for example, Gini index) • Assume two attributes A,B with maximum G • If G(A) – G(B) > ε, then with probability 1-δ, A is the correct attribute to split • Memory needed = O(dvc) (dimensions, values, classes) • Can prove that produced tree is very close to optimal tree.

  43. VFDT Tree • Extension of Hoeffding tree • Breaks ties more aggressively (if they delay splitting) • Computes G after nmin tuples arrive (splits are not that often anyway) • Remove least promising leaf nodes if a memory problem exists (they may be reactivated later) • Drops attributes from consideration if at the beginning their G value is very small

  44. CVFDT System • Source producing examples may significantly change behavior. • In some nodes of the tree, the current splitting attribute may not be the best anymore • Expand alternate trees. Keep previous one, since at the beginning the alternate tree is small and will probably give worse results • Periodically use a bunch of samples to evaluate qualities of trees. • When alternate tree becomes better than the old one, remove the old one. • CVFDT also has smaller memory requirements than VFDT over sliding window samples.

  45. OLAP • On-Line Analytic Processing • Requires processing very large quantities of data to produce result • Usually updates are done in batch, and sometimes when system is offline • Organization of data extremely important (query response times, and mainly update times can vary by several orders of magnitude)

  46. Terminology • Dimension • Measure • Aggregate Function • Hierarchy • What is the CUBE operator: • All 2D possible views, if no hierarchies exist • , if hierarchies exist

  47. Cube Representations - MOLAP • MOLAP: Multi-dimensional array • Good for dense cubes, as it does not store the attributes of each tuple • Bad in sparse cubes (high-dimensional cubes) • Needs no indexing if stored as is • Typical methods save dense set of dimensions in MOLAP mode, and index remaining dimensions with other methods • How to store? Chunk in blocks to speed up range queries

  48. Cube Representations - ROLAP • Store views in relations • Needs to index produced relations, otherwise queries will be slow. • Indexes slow down updates • Issues: If limited size is available, which views to store? • Store fact table, and smaller views (ones who have performed most aggregation) • Queries usually specify few dimensions • These views are more expensive to compute on-the-fly

  49. Research issues- ROLAP • How to compute the CUBE • Compute views from smaller parent • Share sort orders • Exhibit locality • Can the size of the cube be limited? • Prefix redundancy (Cube Forests) • Suffix Redundancy (Dwarf) • Approximation Techniques (Wavelets)

  50. Research Issues - ROLAP • How to speed up selected classes of Queries: Range-Sum, Count…Different structures for each case (Partial Sum, Dynamic Data Cube) • How to best represent hierarchical data. Almost no research here.

More Related