Download
a new model and architecture for data stream management n.
Skip this Video
Loading SlideShow in 5 Seconds..
Aurora PowerPoint Presentation

Aurora

158 Views Download Presentation
Download Presentation

Aurora

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. A new model and architecture for data stream management Aurora

  2. Data Stream Management Why on earth would one need it?

  3. The Problem: Tokyo Traffic Control

  4. Stream Processing for Traffic Control • 24-hour real-time control • 1.000 traffic intersections • 15.154 traffic signals • Input • Cameras • Helicopters • Police • Citizen reports • 17.000 vehicle detectors • Onboard vehicle sensors • Traffic jams, accidents & closed streets • Output • Central monitors • 300 traffic information boards • Digital speed signs • Route signs • Affectors • Adjusted traffic signal lights (7.000) • Communications with officers on site

  5. TTC: Center Display Board

  6. TTC: Information Board

  7. Example Domains • Smart Energy Grid Management • Network Traffic Management • System Monitoring • Road Traffic Monitoring • Military Logistics • Online Auctions • Habitat Monitoring • Immersive Environments

  8. Stream Processing Engines • HADP vs DAHP • Events & Triggers • Continuous Queries • Real-time processing • Transient data • Lossy information

  9. Aurora Overview

  10. The Topic • Aurora • The prototype • DBMS / SPE / DSMS • UI • The query language • The project • The authors

  11. The Authors • M.I.T. , Department of EECS and Laboratory of Computer Science • Michael Stonebraker • Brandeis University, Department of Computer Science • Daniel J. Abadi • Mitch Cherniack • Brown University , Department of Computer Science • Don Carney • Uğur Çetintemel • Christian Convey • Sangdon Lee • Nesime Tatbul • Stan Zdonik

  12. Talk Overview • Stream Processing Engines • SQuAl • Runtime • Related work

  13. Aurora SQuAl (Stream Query Algebra)

  14. SQuAl Overview • Connection Points • Models • Continuous Query • View • Ad-hoc Query • Operators • Order-agnostic • Order-sensitive

  15. SQuAl Operators • Order-agnostic • Filter • Map • Union • Order-sensitive • BSort • Aggregate • Join • Resample • Quirks!

  16. Union (Unordered)

  17. BSort (Ordered)

  18. SQuAl: Example

  19. Aurora Runtime

  20. Query Optimization • Dynamic Continuous Query Optimization • Inserting projections • Combining boxes • Reordering boxes • Ad-hoc query optimization

  21. Real-time Scheduling • Timestamped Tuples • Train scheduling • Interbox nonlinearities • Intrabox nonlinearities • Superboxes • Introspection • Static • Run-time

  22. Handling overload • QoS specifications • Response times • Tuple drops • Values produced • Load Shedding • Not Implemented at the time

  23. Aurora Related work

  24. Related work • STREAM • Stanford University, 2000-2006 • Telegraph • UC Berkley, 2000-2007? • SASE • UC Berkley / Mass Amherst, 2006-2008? • Cayuga • Cornell University, 2005-2007? • PIPES • University of Marburg, 2003-2007? • NiagaraCQ • University of Wiscon-Madison, 1999-2002

  25. Aurora’s Evolution

  26. Complex Event Processing Today • Oracle • Oracle CEP • Microsoft • MS SQL Server StreamInsight • Open Source • OpenPDC • Aleri • Coral8 • TruViso • StreamBase • Aurora’s Grandchild • IBM • SPADE • Active Middleware Technology

  27. Summary • SPEs address different problems • e.g. dynamic realtime monitoring • Data Active, Human Passive • Realtime, transient, even lossy data • Aurora evolved into StreamBase • SQuAl evolved into StreamSQL • Many production-quality alternatives

  28. Filter (Unordered)

  29. Map (Unordered)

  30. Aggregate (Ordered)

  31. Join (Ordered)

  32. Resample (Ordered) • Based on RRDTool’s philosophy? • Paper: • Simple interpolation • Use The Force, Read The Source: • Average • Count • Sum • Max • Min • LastVal