IS4: A Storage System for Organizing Physical Data

1. IS4:A Storage System forOrganizing Physical Data Jorge Ortiz University of California, Berkeley

2. Talk overview • Integrated sensor stream storage system (IS4) • Recap from last retreat • Physical data • Representation of: • Physical devices • Measurements • Environments • Management challenges • Current features and future work

3. Multi-scale Energy Network Wind Modeling IS4 Gen-to-Building Gen-to-Grid Facility-to-Building Building-to-Grid Building-to-Grid Facility-to-Building uGrid-to-Grid Facility-to-Building Grid Storage-to-Building Building MR-to-Building Demand Response Machine Room Load Following Temperature Maintenance Supply Following Plug Loads Web Server Instrumentation Models Grid OS Instrumentation Instrumentation Lighting CompressorScheduling Web App Logic Instrumentation Models Models Models DB/Storage Facilities Routing/Control Control Building OS Load Balancer/Scheduler Fridge as a model Thermal storage system Power-AwareCluster Manager Controls

4. SCADA SystemsSupervisory Control and Data Acquisition + SODA4R787__ART + Data

5. Problems with SCADA • No systematic organization of data andmetadata • Metadata embedded in UI • Difficult to find context information • Incomplete data and metadata • Mostly 0’s • Missing type descriptions, location information

6. Capturing building layout Chiller Pump Zone Chiller Vent Vent Pump SF EF AHU

7. Entity relationships • Building • Systems • Spaces/Zones • Structural relationship • Sensors/Actuators • Inside systems • Inside zones/spaces • Types System Chiller Space Pump Zone Chiller Vent Vent Pump SF EF AHU

8. Building entity schema

9. Relevant queries • List temperature sensors on 4th floor of Soda hall • SELECT id FROM zone_sensors WHERE building=‘Soda’ and floor=4 AND id LIKE ‘%ART’ • Get all temperature sensor data in the last day on the 4th floor of Soda • SELECT * FROM sensor_data WHERE id=[list of sensors] AND timestamp >= ’ 2009-12-16 00:00:00’ and timestamp<=now()

10. System objectives • Physical data storage repository • Append only, forever • Management of data and metadata • Changes occur over time • Physical-data queries • Historical and real time • Security • Secure access to data streams, distillates, metadata

11. What is physical data? • Data collected from the environment • Why is it different from other kinds of data? • Data is produced by sensing devices • Devices are placed in the physical environment • Placement matters • Deployment changes over time • Devices, configuration, environment Data, metadata relationship-management is challenging

12. Sensing device metadata • Make • Dent SmartLogger • Model • CTLogger TOU-CT • url • http://www.dentinstruments.com/media/PDF/SMARTware2009_Manual.pdf • Make • ACme • Model • Revision 1 • url • http://smote.cs.berkeley.edu:8000/tracenv/wiki/ACME

13. Object schema { “type“:"object_stream", “$schema”:”object_stream_schema.json”, “device_name":“dentMeter123", "model":“CTLogger TOU-CT", "desc":“dent meter”, “url”:” http://www.dentinstruments.com/media/PDF/SMARTware2009_Manual.pdf” } instance { "description":"object_stream_schema", "type":"object", "properties": { “type":{"type":"string","optional":false, "options":[{"value":"object_stream"}]}, "device_name":{"type":"string", "optional":"false"}, "model":{"type":"string","optional":false}, "desc":{"type":"string","optional":true}, “url":{"type":"string",”format”:”url”,"optional":true } } schema

14. Communication information • Network address or URL • Function calls, parameter description • Functional schema or url documentation http://coryacme.cs.berkeley.edu/data/33 2001:252:0:1::2008:6 and 2001:252:0:1::2008:8

15. Logic schema { “type“:“logic", “$schema”:”logic_stream_schema.json”, “address”:”http://is4/resource”, “function”:[{ “function_name”:”a”, “opertation”:”GET”, “parameters”:[{“field”, “name”}] } } instance { "description":“logic_stream_schema", "type":"object", "properties": { “type":{"type":"string","optional":false, "options":[{"value":“logic"}]}, “$schema”:”logic_stream_schema.json”, “address”:{“type”:”string”, “format”:[“string”, “ipv4”, “ipv6”, “url”], “optional”:”false”}, “functions”:{“type”:”array”, “options”:[{ “function_name”:”{“type”:”string”}, “operation”:{“type”:”string”, “optional”:”true”}, “parameters”:{“type”:”array”, “options”:[”object”, “string”, “$ref”]} } } } schema

16. Sensor context information Vibration Humidity Temperature Pressure

17. Electric load tree+ Electrical Load Tree Physical CT mains power monitoring Circuit/breaker panel level power monitoring ACme: plug load energy monitor and controller + Slide courtesy of Xiaofan Jiang Humidity Temperature PAR/TSR Vibration

18. Context description • General description • Google JSON GeoCoding object • http://code.google.com/apis/maps/documentation/javascript/v2/services.html#Geocoding_Object • IS4 context graph • Schema that expresses contextual objects and relationships as a graph

19. IS4 Context graph schema Context Node: { "label":{"type":"string", "optional":false}, "name":{"type":"string", "optional":false}, "type":{"type":"string","options":[{"value":"context_node"}], "optional":false}, "description":{"type":"string", "optional":false}, "properties": { "AssociatedDevices":{ "type":"array", "options":[{"value":"string","format":"UUID"}], "optional":true }, "parents":{ "type":"array", "options":[{"value":"string", "properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "children":{ "type":"array", "options":[{"value":"string", "properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "cnid": { "type":[“string"] "description":"local node identifier", "optional":false }}} Context Edge {… "label":{"type":"string", "optional":false, "description":"String with spaces"}, "name":{"type":"string", "optional":false, "description":"String without space"}, … "sourceNode":{"type":[“string"], "description":"local node identifier", "optional":false}, "destinationNode":{"type":[“string"], "description":"local node identifier", "optional":false}, "AssociatedDevices":{ "type":"array", "options":[{"value":"string","format":"UUID"}], "optional":true }} schema

20. Cory hall load tree Load Tree Image Graphical representation

21. Context state snapshots Time t1 Time t2

22. Queries • What floor consumes the most power per month? • Send alert if the air conditioner in room 465 consumes more than Y kW in a 24 hour period • Send alert if independent zones become dependent • What is the most common state of the air/water flow graph?

23. Expressing physical-data queries • Semantic graph + temporal data • Physical and logical relationships • Data streams associated with graphical elements • Easily express queries across both data types • Efficient storage • Efficient access

24. IS4 Current Status • Data publishers identified by unique id (and IS4 resource URI). • Publishing process includes object and logic JSON objects as input • Context data managed separately • Publisher may belong to multiple contexts • Data and metadata stream explicitly bound

25. Current Architecture Publish CT mains power monitoring Subscribe HTTP/REST Subscribe Manager Publish Manager Proxy Manager Storage Context Manager ACme: plug load energy monitor and controller Humidity PAR/TSR Temperature Vibration

26. IS4 Interface Overview / #root – status information [GET] /is4 # a particular channel [GET] /info # statistical system information [GET] /publish # publishing resource [GET/PUT] /all # list of all publishers [GET] /id # sub-children [GET] /<id> # id of publisher [GET/POST] /name # name alias for this id [GET] /mysubs # list/add to pub subscriptions [GET/PUT/DELETE] /metadata # returns publisher-associate metadata [GET] POST requests supply JSON objects as arguments: POST: http://is4.cs.berkeley.edu/is4/publish/id/123456/mysubs?username=jortiz&pw=8832749823{“SubscribeTo":[13241324,234213,45456745], “UnsubscribeFrom":[45432,567365,21465765,2435786] } Username and pw hash Publisher Id

27. Ongoing work • Security • Token-based security • Tokens encapsulate access rights to resources • Resources referenced by IS4 URI • Permissions: GET, PUT, POST, DELETE • GET = read • PUT = create • POST = update • DELETE = delete • Query interface • Semantic graph + timeseries

28. Future Work • Incorporate security model into current implementation • Move the data store into Amazon cloud • 14 Dents, 1 min period = ~15 MB/hour  130 GB/year (uncompressed) • Data processing functionality • Interpolation, extrapolation, other time series processing • JSON streaming query engine

29. Summary • Current building management systems poorly handle physical data • Physical data is tightly bound metadata and context • IS4 provides schemas to express metadata • IS4 implements mechanisms to manage data/metadata binding

30. Thank you • More information • http://smote.cs.berkeley.edu:8000/tracenv/wiki/is4 • Jorge Ortiz • jortiz@cs.berkeley.edu • Questions?

31. Extra Slides

32. Snapshots of physical information • Time t1 • Time t2

33. Publishers and Subscribers

34. Open design issues • Doesn’t PowerMeter already do (some) of this? • They also deal with the data stream; context is not used • How different from pubsubhubbub? • Explicit context management

35. Interpreting data across contextual changes • All the data is timeseries • Real time and historical querying • User can be notified if context changes have occurred over query-time interval

36. General notes • Talk about what I presented at the last retreat • Include previous work • Include a summary slide at the end • Update the opening slide (missing sponsors) • Look at Stephen’s slides and figure out how to incorporate them • You have to make connections between the two • Ask Gil for his slides to see how to segway these into that

37. General notes (2)