Challenges in Sensor Network Query Processing

1. Challenges in Sensor Network Query Processing Sam Madden NEST Retreat January 15, 2002

2. Outline • Background • Server Side Solutions • Fjords, Sensor Proxies, CACQ • Sensor Side Solutions • Catalog Management • Aggregation • Future Work

3. Background: Query Processors

4. What is a Query? • Declarative statement requesting a subset of data • Possibly transforming or computing statistics about that data • Data independent • Query can apply to any data

5. What is a Query Processor? • Converts declarative queries into flow of data operators, a query plan • Relational Operators: • Project, Select, Join • ‘Scans’ read data from base relations, indices, etc. • Traditional Flows: • Pull based, ‘iterator model’ • Higher level operators call ‘getNext()’ to extract data from lower level operators

6. Query Optimizer • Given a declarative query, build the ‘best’ query plan • Choose which operators to run • What order to run them in • Where to run them • In distributed databases

7. Why Databases and Sensors? • All applications depend on data processing • Declarative query language over sensors attractive • Application specific solutions difficult to built and deploy • Want “to combine and aggregate data streaming from motes.” • Sounds like a database…

8. New Problems In Sensor Databases • Sensors unreliable • Come on and offline, variable bandwidth • Sensors push data • Sensors stream data • Sensors have limited memory, power, bandwidth • Communication very expensive • Sensors have processors • Sensors very numerous

9. Components of A Sensor Database • Server Side • Query Parser • Catalog • Query Optimizer • Query Executor • Query Processor • Sensor Side • Catalog ‘Advertisements’ • Query Processor • Network Management

10. Outline • Background • Server Side Solutions • Fjords, Sensor Proxies, CACQ • Sensor Side Solutions • Catalog Management • Aggregation • Future Work

11. Fjords • Query Plan Abstraction to handle lack of reliability and streaming, push based data • Combine push and pull in arbitrary combinations • Use connectors between operators to isolate them from flow direction • “Bracket Model” – Graefe ‘93

12. Fjords (Continued) • Operators assume non-blocking queue interface between each other. • Queues implement push vs. pull • Pull from A to B : Suspend A, schedule B until it produces data. A cannot go forward until B produces data. • Push from B to A : A polls, scheduler thread invokes B until it produces data. A can process other inputs while waiting for B. • Supports parallelism between operators via queues, state machines, and OS (e.g. NIC buffers, DMA) in operator transparent way.

13.   Fjords Example Push Push  Pull Samuel Madden, Michael J. Franklin. Fjording The Stream: An Architecture For Queries Over Streaming Sensor Data. International Conference on Data Engineering, 2002. To Appear, Feburary 2002.

14.   Fjords Example Push Push  Pull Samuel Madden, Michael J. Franklin. Fjording The Stream: An Architecture For Queries Over Streaming Sensor Data. International Conference on Data Engineering, 2002. To Appear, Feburary 2002.

15.   Fjords Example Push Push  Pull Samuel Madden, Michael J. Franklin. Fjording The Stream: An Architecture For Queries Over Streaming Sensor Data. International Conference on Data Engineering, 2002. To Appear, Feburary 2002.

16.   Fjords Example Push Push  Pull Samuel Madden, Michael J. Franklin. Fjording The Stream: An Architecture For Queries Over Streaming Sensor Data. International Conference on Data Engineering, 2002. To Appear, Feburary 2002.

17.   Fjords Example Push Push  Pull Samuel Madden, Michael J. Franklin. Fjording The Stream: An Architecture For Queries Over Streaming Sensor Data. International Conference on Data Engineering, 2002. To Appear, Feburary 2002.

18.   Fjords Example Push Push  Pull Samuel Madden, Michael J. Franklin. Fjording The Stream: An Architecture For Queries Over Streaming Sensor Data. International Conference on Data Engineering, 2002. To Appear, Feburary 2002.

19.   Fjords Example Push Push  Pull Samuel Madden, Michael J. Franklin. Fjording The Stream: An Architecture For Queries Over Streaming Sensor Data. International Conference on Data Engineering, 2002. To Appear, Feburary 2002.

20.   Fjords Example Push Push  Pull Samuel Madden, Michael J. Franklin. Fjording The Stream: An Architecture For Queries Over Streaming Sensor Data. International Conference on Data Engineering, 2002. To Appear, Feburary 2002.

21.   Fjords Example Push Push  Pull Samuel Madden, Michael J. Franklin. Fjording The Stream: An Architecture For Queries Over Streaming Sensor Data. International Conference on Data Engineering, 2002. To Appear, Feburary 2002.

22. Fjords Applications • Combine traffic streams with web-based accident reports Francis Li, Sam Madden, Megan Thomas. Traffic Visualization. http://www.cs.berkeley.edu/~mct/infovis/project/traffic.html

23. Operators for Streaming Data • Need special operators for dealing with streams (See P. Seshadri, et al. The design and implementation of a sequence database systems..VLDB ’96) • In particular, streams can’t be joined or sorted in the traditional sense • Solution: Use windows – e.g. “Zipper Join”

24. Sensor Proxy • Energy-sensitive database operator • Buffer sensor tuples and route to multiple user queries to hide query load from sensors • Push aggregation operators into sensors to reduce communications load • Dynamically adjust sample rate based on user demand • Push results into Fjords so that other operators don’t block waiting on slow or dead sensors

25. Some Results • Pushing predicates into sensors can vastly reduce costs: Atmel Simulator 100 samples / sec 5 vehicles / sec 7x power savings

26.   Query 1 Query 2   Stocks. symbol = ‘APPL’ Stocks. symbol = ‘MSFT’ Stock Quotes ‘MSFT’ ‘APPL’ Stock Quotes CACQ • Expect hundreds to thousands of queries over same sensor sources • Continuously Adaptive Continuous Queries • Continuous Queries: Long running queries which combine selections and joins to improve efficiency (See Chen, NiagaraCQ, SIGMOD 2000)

27. static dataflow eddy CACQ (Cont.) • Continuous Adaptivity From Eddies • Route tuples differently, depending on selectvity and cost estimates of operators Diagrams Courtesy Joe Hellerstein

28. CACQ (cont.) • Combining CA with CQ is a win: • CQ increases number of simultaneous queries • Adaptivity well suited to long running queries • Eddies allow us to avoid ugly query-optimization phase in traditional CQ • Eddies + Streams == few copies, unlike traditional CQ

29. CACQ (cont) Look for a paper in SIGMOD 2002 (fingers crossed!)

30. Outline • Background • Server Side Solutions • Fjords, Sensor Proxies, CACQ • Sensor Side Solutions • Catalog Management • Aggregation • Future Work

31. Sensor Side Solutions • CACQ + Fjords provides interface + performance on QP, but sensors still need help: • Locate / identify sensors • Reduce power consumption • Take advantage of processors? • Improve responsiveness

32. Cataloging Sensors • To query sensors, need a way to locate, identify properties, extract values • Goal: Drop a bunch of sensors around the DBMS, allow them to be queried without manual effort • Idea: Add a layer to each sensor which advertises its capabilities

33. #temperature sensor field { name : "temp" #optional type : int units : celsius min : -20 max : 100 bits : 8 sample_cost : 10.0 J #optional -- for use in costing sample_time : 10.0 ms #optional -- for use in costing input : adc2 #optional : read from adc channel 1 sends : ondemand accessorEvent : GET_TEMPERATURE_DATA responseEvent : TEMPERATURE_DATA_READY } Compiled in 27 bytes of memory Layer to register with Query Processor Can be “push” or “pull” Catalog (Continued)

34. Aggregating Over Sensors • Sensor Proxy combines user queries, pushes down aggregates • Goal: Save energy, increase efficiency • Idea: Take advantage of the routing hierarchy

35. Why bother with aggregation • Individual sensor readings are of limited use • Interest in higher level properties, e.g. what vehicles drove through, what is the spread of temperatures in the building • We have a processor & network on board, lets use it • We cannot survive without aggregation • Delivering a message to all nodes much easier than delivering a message from each node to a central point • Delivering a large amount of data from every node harder still, vide connectivity experiment • Forwarding raw information too expensive • Scarce energy • Scarce bandwidth • Multihop performance penalty

36. Aggregation challenges • Inherently unreliable environment, certain information unavailable or expensive to obtain • how many nodes are present? • how many nodes are supposed to respond? • what is the error distribution (in particular, what about malicious nodes?) • Trying to build an infrastructure to remove all uncertainty from the application may not be feasible – do we want to build distributed transactions? • Information trickles in one message at a time • Never have a complete and up-to-date information about the neighborhood • What type of information should we expect from aggregation • Streams • Robust estimates

37. What does it mean to aggregate(The DB Perspective) • General purpose solution: apply standard aggregation operators like COUNT, MIN, MAX, AVERAGE, and SUM to any set of sensors. • Existing solutions are application specific • In sensors, operators may be arbitrary signal processing functions • By assuming a standard interface, many optimizations are possible • Example: TopN queries via hypothesis testing • Provide grouping semantics: e.g. ‘select avg(temp) group by trunc(light/10)’ • In sensor networks, groups may be random samples t1 t2 t3 t4 t5 t6 t7 t8 t9

38. Outline • Background • Server Side Solutions • Fjords, Sensor Proxies, CACQ • Sensor Side Solutions • Catalog Management • Aggregation • Future Work

39. Future Work • DBMS Side • Efficient Catalog Management • Moving Object Databases • Query Optimization Techniques • Sensor Side • Efficient Grouping • Joins over Network Topology • Non Standard Aggregate Functions • Somewhere In Between • Histograms and other Correlations • Sampling and Compression for Streams • Real Query Language / API • Demonstration Apps (SIGMOD Demo)

40. Questions?

41. 3 4 5 1 2 Scenario: Count

42. 1 2 3 4 5 Sensor # Time Goal: Count the number of nodes in the network. Number of children is unknown. Scenario: Count

43. 1 2 3 Sensor # Time Goal: Count the number of nodes in the network. Number of children is unknown. Scenario: Count

44. 1 2 3 4 Sensor # Time Goal: Count the number of nodes in the network. Number of children is unknown. Scenario: Count

45. 1 2 3 4 5 Sensor # Time Goal: Count the number of nodes in the network. Number of children is unknown. Scenario: Count

46. 1 2 3 4 5 Sensor # Time Goal: Count the number of nodes in the network. Number of children is unknown. Scenario: Count

47. 1 2 3 4 5 Sensor # Time Goal: Count the number of nodes in the network. Number of children is unknown. Scenario: Count

48. 1 2 3 4 5 Sensor # Time Goal: Count the number of nodes in the network. Number of children is unknown. Scenario: Count

49. Counting Lessons • Take advantage of redundancy to improve accuracy (reply to all parents, not just one) • Use broadcast to reduce number of messages • Result is a stream of values: much more robust to failures, movement, or collision than a single value.

50. Aggregation in network programming • Network programming problem • Reliable delivery of a large number of messages to all nodes in range, while exploiting the broadcast nature of the medium • Basic setup • Broadcast a known number of idempotent program fragments • Each node keeps a bitmap of fragments received (1=packet received) • Two stages of the problem: single hop, and multihop • Solutions • Single hop, dense cell • Broadcasting the program – trivial, the central node broadcasts • Feedback from nodes – broadcast a request from the central node: Is anyone missing packets in this packet range? • Convergence: no replies to the request