Bridging the Gap between Humans and the Physical World

1. Bridging the Gap between Humans and the Physical World A Step Toward Reducing Energy Consumption and Increasing Comfort Dr. Fred Jiang / 姜小凡 In collaboration with Mike Liang, Jeff Hsu, Caiquan Liu, Jie Liu, and Feng Zhao Mobile and Sensing Systems Group Microsoft Research Asia National Taiwan University / 2011-4-1

2. Motivation • We interact with our environment in very limited ways • Simple time series data collection • Actuation is limited because of lack of uniform interface • Physical objects cannot proactively reach out to us • Result: waste in energy and suboptimal comfort • Example 1: environment (physical objects) aware of who is acting on it, collect usage / energy data in real-time(personal energy footprint) • Example 2: alert me if something is not normal • Example 3: objects interacting with objects

3. Related Work • Ambient intelligence • Roomware • Smart objects • Localization • X10 • Beaver monitoring • Ubicomp literature

4. We Propose • Bi-direction interaction and information flow between humans and physical objects • For humans to have real-time finer-grained knowledge and control over physical objects • Environment to be aware of humans, and make intelligent decisions, proactively and reactively • Bridging the gap between humans and their environment • Our vision of IoT (it’s not about simply putting a networking interface on everything)

5. Three Primary Components • Precise indoor localization primitive • Reliable IPv6 based networking • Uniform interface for abstracting services of physical objects

6. Virtual Overlay Entrance to 4th floor name: fridge states: on : true temp : 5 door : open commands: turn_on() turn_off() events: notifyDorOpn notifyPwrUsg children: []

7. Indoor Localization • Previous work • What we need • Challenges

8. Localization Options • Wifi radio • BT radio • Inertial sensor • Optical (Kinect) • RFID • Magnetic (PKE)

9. Magneto-Induction Slide courtesy of Andrew Markham (http://www.comlab.ox.ac.uk/people/andrew.markham/)

10. Magneto-Induction Slide courtesy of Andrew Markham (http://www.comlab.ox.ac.uk/people/andrew.markham/)

11. Magneto-Induction Slide courtesy of Andrew Markham (http://www.comlab.ox.ac.uk/people/andrew.markham/)

12. Challenges for Indoor • 4m x 1m antenna -> 5m^3 tracking • We don’t have that much space! • We use a 8cm x 1.5cm transmitter antenna • And 3D coil IC (2cm x 1cm) as receiving antenna • Power supply issues • Outdoor vs indoor • Interference issues • Phone magnetometer (digital compass)

13. Microsoft LivePulse

14. 125kHz MI Transmitting Antenna Mechanical Relay MI MOSFET Driver 900MHz PCB Antenna CC430 SoC Energy Meter IC Hall-effectSensor Mains Power AD/DC/DC Converstion 90-240VAC -> 12VDC -> 5VDC -> 3.3VDC

15. Microsoft LivePulse • Wireless energy monitoring • Precise indoor localization Range: ~ 3m omnidirectional ~10cm dropoff zone Consistent over time Robust against interference and multipath Block by metal

16. Microsoft LiveLink Wakeup 3D MI Receiving Antenna

17. Networking • Requirements • Reliable • Globally addressable • Scalable • Low power

18. Radio Consideration • 802.15.4 • 2.4GHz • 863 MHz to 928MHz • 100-150m range • US • 802.15.4c (c for china) • 779-787 MHz • 250m range • Core module choices • 60USD Epic core • 20USD SuRFcore • Energy

19. Networking Overview JSON / RESTful services over Internet Repository of virtual objects One-hop IPv6 communication over 802.15.4c

20. Virtual Object Abstraction Layer apartment bedroom office bathroom hallway desk desktop room portable lamp computer lamp heater coffee laptop piano pinball machine computer keyboard machine • Hierarchical • Event based LCD computer monitor tower

21. Object Representation • States • E.g. On/off • Commands • E.g. turn on / read power / set temp / count people • Events • E.g. door opened / light-off / over-threshold • Children • Ptr to children objects

22. 1. DISCOVERY_REQ to http://<ipv6OfOffice> 2. DISCOVERY_REPLY name: office states: lightOn? : True doorOpen? : False roomOccupied? : True roomTemp? : 30C roomEnergy? : 200W commands: toggleLight numOccRoom lastTimeDoorOpened events: notifyDoorOpen notifyNewOccupant notifyHourlyEnergyUse children [ipv6]: coffeeMachine deskLamp laptopComputer desktopComputer pianoKeyboard roomLamp pinballMachine portableHeater 3. EVENT_SUBSCRIPTION [notifyNewOccupant] “let me know if someone enters room” 4. NOTIFICATION “someone just entered your room”

23. Open Source Standard • JSON is the object encoding • SOAP / WSDL / Zigbee SE Profile – Nooo • REST/HTTP is the interchange scheme • HTTP_GET / HTTP_GET / HTTP_RESPONSE • IPv6 • Interoperate with existing IP devices without understanding additional protocols

24. Applications • Personal energy footprint • Social networking • Health care

25. Related Efforts • Smart meters (via utilities) • Enables time-of-day billing • Google PowerMeter • Visualization of whole-house usage • Defining a API for energy data • Microsoft Hohm • Complimentary work (help learn signatures) • House-level / NILM / can benefit from DB of appliance signatures • In the right direction, but not enough • Only at the whole-house level • Lack of actuation / control Source: http://www.microsoft-hohm.com Source: X. Jiang Dissertation

26. Individualized Energy Feedback and Control • Real-time energy apportionment • Individualized energy accounting / billing • Energy map / trail of occupants • Alerts and abnormally detection • Via cellphone • Remote alerts • Control and actuation • Automated control of devices • Scheduling

27. An Energy-Centric Ecosystem – Social Network of Energy • Social network based on real-time individual energy usages • Windows 7 Gadget • Windows Mobile 7 App • Facebook app • Reward systems • Building managers have fine-grain control and view

28. mobile application

29. web application - synergy

30. Social Network of Energy • Localization • Location Status updates, kind of like Foursquare Checkins. • “Mike is now in the coffee room.” • Apportionment • Now that we have localization primitive, we can specifically attribute the use of public energy to individual users. Very useful. • “Attribute the last 3 minutes of coffee energy usage to Jeff.” • Add this to personal monitoring of private appliances, and we now have a complete carbon footprint.

31. web application dashboard - map

32. Social Network of Energy • Map of User’s owned areas…and where energy usage is taking place the most • Social Interaction • Compare Energy Usage of Specific Appliances, or aggregate • Games • User VS. User [One-Month Challenge, percent improvement?] • Team VS. Team

33. web application dashboard - energy

34. Social Network of Energy • Add Appliances [Home and Work] • Energy Usage Trends • Day to Day Comparisons in a table • Energy Usage Graphs [Compare different devices] • Email Summary of Energy Usage to User [each day, or week, or month]

35. web application – user profile

36. Social Networking “Wine Party” App • Frequently, it is very difficult to meet people you click with at social gatherings. • Ex. You are at a company party. You are bored. • You pull out your mobile phone. A map of the room you are in shows up, and there are markers on the map. • Click on a marker, and you can see that person’s interests, details, hobbies, etc. • See twitter feed, facebook profile, etc. • When someone who match your interest pass by, your phone vibrates to alert you.

37. Health Care • Room is aware of status of elderly occupants • 1. phone insert trigger / event • 2. object representing the room • 3. room send a signal to alert

38. Conclusion • Bi-direction interaction between humans and physical objects • For humans to have real-time fine-grained knowledge and control over physical objects • Environment to be aware of humans, and make intelligent decisions, proactively and reactively • Precise indoor localization primitive • Reliable IPv6 based networking • Uniform virtual representation of physical objects

39. Thank you • Questions? • Feel free to contact me at fxjiang@microsoft.comhttp://research.microsoft.com/people/fxjiang