Summer School: Events, Publish/Subscribe & Systems

1. http://canoe.ifi.uio.no/index.php/about-summer-school Summer School: Events, Publish/Subscribe & Systems Hans-Arno Jacobsen On Behalf of the Organizers: Frank Eliassen & Roman Vitenberg

2. VElkommen

3. For Your Information • Internet access • Breakfast, lunch, dinner and breaks • 7 AM – 9 AM (Breakfast) • 12:30 (Lunch everyday including Friday) • 7:30 - 9 Monday • 6:30 - 8 Tuesday & Thursday • Viking Experience: Monday 6 PM at the reception • Wednesday afternoon boat trip (dinner included) • Leave the hotel at 3:45 • Leave in front of the city hall in Oslo at 9:15

4. Around the Sundvolden Hotel

5. Quick Introductions • In one minute, please tell us • Who you are • From where you are • Your main research interests • What you expect from the week • Where you heard about the Summer School (DBWORLD, EPTS, TCCC, etc.)

6. CANOE • Canada-Norway Partnership in Higher Education 2008-2011 • http://canoe.ifi.uio.no/ • Collaborate in research and teaching on • Distributed systems, networking, and event processing • Jointly organize workshops, summer schools, student exchanges, and research visits

7. Summer School Objectives • Learn about events, publish/subscribe and systems • Interact on the above topics in discussions, brainstorming sessions, and poster session • Network and have fun

8. Collaborate & Provide Feedback • Collaborate Wiki • http://debs.msrg.utoronto.ca/collaborate • Login via … • Anonymous feedback form • Feedback URL • Please use as much and as often as you’d like • Many of the questions may only apply towards the end of the week • Session specific forms available via the Collaborate Wiki (login required)

9. The Program

10. Organization & Structure • Lectures and tutorials • Listen, learn and ask • Panel, discussion and Q/A session • Ask and participate • Break-out sessions and social events • Engage, collaborate, and contribute

11. Events, Publish/Subscribe & Systems • Events • Are the problem • Publish/Subscribe, Streams, Rule Engines, Event Processing • Are the abstractions & the paradigms towards a solution • Systems • Are the tools to build viable solutions

12. Applications A E B C F D Supply chain and logistics Workflows, business processes and job scheduling Job A done Shipped Triggered Delivered Fault Ordered Event-Based Callback received Razor SKU Loan service invoked Light on Transformed Temperature high Service oriented architecture RFID and sensor networks

13. Other Applications Algorithmic & automated trading Asset tracking Audit and compliance checking Automated security pricing Business activity monitoring (BAM) Business intelligence (BI) Business process execution Business activity monitoring Capital marker surveillance Content distribution (stock) Fraud detection and prevention Intrusion detection Information dissemination & filtering Location-based services Luggage handling at airports Network management Order management and advanced pricing Operational risk assessment RFID-based systems Resource management Sensor networks System monitoring and diagnostics (alphabetical order) Extracted from a series of use case presentations at EPTS meetings in 2007-present.

14. Observations Events in many verticals Automotive Banking & Finance Insurance Health-care Transportation Military … Core functions required Monitor and observe Detect Filter Correlate Infer and predict Disseminate

15. Event Processing Initiatives ACM Distributed Event-based Systems Conference (DEBS) http://www.debs.org/2009 Event-based Research Portal http://www.event-based.org Complex Event Processing http://complexevents.com/ Event Processing Technical Society (EPTS) http://www.ep-ts.com

16. Our Program

17. Our Program Monday • Towards a discipline of “smart” systems by Mani Chandy, Caltech • Pub/Sub at Google by John Reumann, Google Tuesday • Discussion & Q/A with John Reumann • Commercial Trends in Event Processing by Paul Vincent, TIBCO

18. Our Program Tuesday (cont.’d) • Poster presentations with all of us • Event Processing with the PADRES Pub/Sub System by Hans-Arno Jacobsen, University of Toronto • Break-out sessions with all of us

19. Our Program Wednesday • High-performance Event Stream Processing Infrastructures and Applications by BugraGedik, IBM • Real-time Data Distribution by Angelo Corsaro, PrismTech • The Next Generation Application Server – How Event-based Processing Yields Scalability by Guy Korland, Gigaspaces • Field Trip

20. Our Program Thursday • Content Routing and Filtering in the Large Scale, Pascal Felber, University of Neuchatel • Discussions and Reflections with Mani Chandy • Mohammed and the Mountain: Distributing the Data or Moving the Data by Eric Jul, University of Copenhagen • Panel: Streams vs. Rules vs. Subscriptions by BugraGedik • Break-out sessions

21. Our Program Friday • Scalable Hostting of Web Applications by Pierre Guillaume, Free University Amsterdam • Break-out Session Summaries & Results with all of us

22. Poster Session

23. Poster Session: Tuesday @ 2-3:30 • Abstracts are posted on our web page • Please set up just after lunch (around 1:45) • We may use walls with hotel provided tape in the conference room • We start the session with one minute pitches by authors (alphabetical order) • Please line-up at the front (see web site) • Pitches are performed without slides

24. Break-out Sessions

25. Break-outs Organization • Meet in small groups during two slots throughout the week (evening slots on Tuesday and Thursday) • Adjourn in plenum on Friday to present discussion and results • Each group is given some guiding questions, but may define their own • Groups should be self-organizing, i.e., designate a scribe and a presenter for Friday • http://debs.msrg.utoronto.ca/collaborate

26. Suggested Discussion Groups • Simulation & evaluation • Applications • Research agenda & road map • Marketing & advocacy • Practical limitations • Collaborations • Interoperability • Management issues We would like to charter at least 6 groups. Others ? We will discuss the details in the plenum on Tuesday evening at 8 PM, followed by the group discussions.

27. Discussions

28. Discussion Sessions • Tuesday morning with John Reumann about Events, Pub/Sub and Systems Research Challenges at Google • Thursday morning with Mani Chandy as Discussion and Reflections about What we Heard and Learned so far • Thursday afternoon with BugraGedic as panel discussion on Streams vs. Rules vs. Subscriptions

29. Fundamental Questions • What abstractions support these core functions? • What models, formalisms techniques, algorithms underlie these abstractions & core functions? • How are these abstractions used? • What abstractions can best be used to build event processing requiring applications? • Databases? • Messaging middleware? • Transaction processing systems? • ... • Is there a need for developing an Event Management System? In analogy to a Database Management System for data.

30. My Advice • A Summer School is not a conference • Please ask questions, any questions • We want to learn from each other • Bring yourself in

31. What is an event? • Etymology • Latin: eventus; • evenire to happen; • e- + venire to come

32. Various Interpretations A postulated outcome, something that follows as a result or consequence, condition, or eventuality; “in the event that I am not there, you may …” Something that happens, a noteworthy happening, an occurrence Any of the contests in a program of sports, a social occasion or activity, a world event The fundamental entity of observed physical reality represented by a point designated by three coordinates of place and one of time in the space-time continuum postulated by the theory of relativity A subset of the possible outcomes of an experiment (probabilities)

33. Event – My Working Definition An event is an instantaneous and asynchronousstate transition in the environment of interest to applications, systems, users. Event versus eventoccurrence Like type or class versus instance In my definition events do not extend over time Interval events are defined by starting and ending events (a.k.a. composite events) My definition does not capture world events Maybe important for a theory of events, but less so for the effective processing of events My definition does not capture interval events Events associated with a time span, but those can be modeled as composite events

34. Synonymously Used The terms Event Event occurrence (Event) message Event object are often used synonymously in practice. Also the pattern for detecting an event is often simply referred to as event.

35. Event source Event producer Event generation Event publisher Sink Consumer Capture Subscriber Similarly, …

36. Event Processing • Event Processing is an emerging area and discipline that aims to define and develop • Abstractions • Architectures • Systems • Languages • Patterns • Models • Standards for the processing of events in systems, applications, and by end users.

37. A Note on CEP Complex Event Processing (CEP) Event processing must not be complex Or else, a marketing nightmare, I suspect  Meant was really Complex-event Processing Not the complex processing of events Let’s just keep to Event Processing and try to make it as simple as possible, 

38. A.k.a. • Event Processing • Complex Event Processing & CEP • Business Event Processing • Intelligent Event Processing • Event Stream Processing Event Processing & CEP are widely used, as opposed to some of the other terms. I am sure there will be more. Marketing 

39. Event Processing Models (1970-present) • Rule-based event processing • Active databases • Publish/Subscribe • Event stream processing (ESP) • Programming language embedded & library

40. Event Processing Languages • Rules • ECA rules • Database triggers • Subscriptions • XPath • SQL with stream processing extensions • Graphical notations

41. Dissemination & Filtering OMG DDS WS Notifications WS Eventing INFO-D AMQP CORBA Events & Notifications APIs & Formats CEI & CBE WSDM & WEF Languages StreamSQL OMG Production Rule Representation (PRR) and W3C RIF OMG Event Metamodel and Profile (EMP) Emerging Standards

42. EP Initiatives ACM Distributed Event-based Systems Conference (DEBS) http://www.debs.org/2009 Event-based Research Portal http://www.event-based.org Complex Event Processing http://complexevents.com/ Event Processing Technical Society (EPTS) http://www.ep-ts.com

43. Events in Industry I • Mostly pre-standardization today • Pre-competitive collaborations • Efforts revolve around activities of EPTS • Gathering of use cases & requirements • Definition of glossary (vocabulary) • Reference architecture • Event processing meta model • Event processing language • In some related areas standardization started

44. Events in Industry II • Predominant trends • Stream-based processing of events through extensions of SQL • Most dominant database vendors • Rule-based processing of events through various rule formalisms • Many traditional middleware vendors • Programming language embedded • Scripted and proprietary There is no one dominant approach today.

45. Events in Industry III • Relational algebra, SQL, and SQL extensions • EP uses and builds on stream processing efforts • Industry adoption of SQL is well known • Industry adoption of SQL extensions is not clear at this state; no de facto standard yet either • Rules • Re-emerging interest in rule-based processing • Intense recent efforts in standardizing • Flexible rule markup languages • Rule interchange formats • Industry adoption of standards is not clear at this stage Predication: We will see many more language dialects until in a few years time a de facto standard emerges. Most likely the approach taken by the surviving or dominant player.

46. Events in Industry IV • Efforts underway at (alphabetical) • IBM, Oracle (& BEA), Tibco, • and many more, sorry for omissions • Start-ups (alphabetical) • Coral8, EventZero, StreamBase • and many more, sorry for omissions There was/is intense activity – let’s hope recent world events don’t interfere.

47. Events in Academia I • Research on events in various sub-areas of computer science and others • Databases • ECA rules & trigger management • Stream processing • Publish/Subscribe • Networking • Event correlation and network management • Systems • Publish/Subscribe • SEDA (staged event-driven architectures)

48. Distributed systems Event distribution and dissemination Content-based routing Publish/Subscribe Software engineering Publish/Subscribe Embedded systems Formalization of event-driven systems Event-driven programming Programming languages Programming with events Events and types Philosophy Notion of events Events in Academia II • Events are broadly • recognized. • There is no unifying theory • of events.

49. Events in Academia III • The DEBS Conference aims to establish a forum that brings together disparate research revolving around events. • Started in 2002 as workshop until 2006 • From 2007 until present an independent conference. • Obtained ACM sponsorship in 2009. ACM International Conference on Distributed Event-based Systems.

50. Data management Relational algebra & theory SQL Started decades ago 35+ year strong research community (VLDB, SIGMOD, ICDE et al.) Industry community Several key players and many players catering to niches Many textbooks Event processing Requires a foundation & theory … a common language Recently started Young and vibrant research community that’s growing (DEBS) EPTS Interest from key players Many small emerging players Essentially none (emerging) Data Management versus Event Processing