1. EasyLiving:Technologies for Intelligent Environments Barry Brumitt, Brian Meyers, John Krumm, Amanda Kern, Steven Shafer

2. Talk Outline EasyLiving Project Overview Critical Technologies Demo Applications

3. Ubiquitous Computing Desktop Computing Mobile Computing Intelligent Environment Disaggregated Computing, Invisible Computing, Pervasive Computing, Ubiquitous Computing

4. (Jeff) User Interface Resource Management Sensing & Modeling (Printer) (Mona Lisa) Ubiquitous Computing Elements People Abstractions Applications & Behaviors Abstractions Computation Things Devices

5. The EasyLiving Project • EasyLiving is a prototype system for building intelligent environments • Key Technologies • Middleware • Perception • World modeling (geometry!) • 3 researchers, 3 SDEs

6. Talk Outline EasyLiving Project Overview Critical Technologies • Middleware • Perception • Geometry Demo Applications

7. Ubiquitous Computing Needs Middleware • Simplified application development • Resource management across disparate heterogeneous devices • Facilities for handling network partitioning (esp. mobile devices)

8. InConcert (now) • Messages passed asynchronously • Messages marshalled into XML • Machine-independent addressing …also being used by other projects at MSR

9. InConcert(coming attractions) • Discovery of new services • Attribute-based service description • Eventing (filters, etc.)

10. Talk Outline EasyLiving Project Overview Critical Technologies • Middleware • Perception • Geometry Demo Applications

11. Perception for Ubiquitous Computing • Who is in the room? Where? • What is Joe doing? • Where is Joe looking? • What is in the room? Where? • What devices are accessible to Joe? • What is Joe pointing at? • Who is speaking now? Typing now?

12. Person TrackingSystem

13. Person Tracking Example More cameras = more coverage (video: 02:30 – 04:30)

14. Problem Dimensions(1) Speech (for example) talking speed dictation continuous speaker speaker dependent speakerindependent Person Tracking number of people in view 1 2 3 n number of cameras 1 2 3 n

15. Problem Dimensions(2) identity none maintain recognize occlusion 0% 100% 50% postures stand stand & sit stand, sit, lay show one image canned sequence video tape live demos always on naïve user real time

16. Keyboard Tracker Rule: Route keystrokes to the session of the person in front of the keyboard. Green dot indicates centroid of keyboard

17. Talk Outline EasyLiving Project Overview Critical Technologies • Middleware • Perception • Geometry Demo Applications

18. What is Geometry? • Euclid’s Five Postulates • A straight line segment can be drawn joining any two points • Any straight line segment can be extended indefinitely in a straight line • Given any straight line segment, a circle can be drawn having the segment as the radius and one end point as center • All right angles are congruent • If two lines are drawn which intersect a third in such a way that the sum of the inner angles on one side is less than the two right angles, then the two lines inevitably must intersect each other on that side if extended far enough. A pervasive mechanism for declaring, representing, and querying the physical relationship between people, places, devices, and things.

19. Geometric Model Benefits • Abstraction Layer • Shared Metaphor with User • Physical Parameters for UIs

20. Other Approaches to Location Context • Semantic tags • Outdoor Beacons: GPS/mobile-phone • Indoor Beacons: RF, IR, ultrasonic • Single coordinate frame • Poor mapping to semantics • Network Routing (connectivity=colocation) • Avoids perception perils • Confusing semantics

21. pda computer display y y y y y y y y map y y y x x x Display 42 DQ x x x x x x x x person camera Dy uncertainty Vision Module badge Dx GPS beacon Measurements Entities Measurement Graph Model Fundamentals • Explicit uncertainty representation • No “favored” coordinate system or origin • Geometry only, minimal semantics

22. y y y y y x x x x x Queries • 2 Fundamental query types • Entities’ relationship • Region membership

23. Geometry(Coming attractions) • Scalability & performance • Integration with semantic location tags • Maps / floor plans, path planning • Geometric discovery

24. Talk Outline EasyLiving Project Overview Critical Technologies Demo Applications

25. Turning On The Lights Methods • “Turn on that light” [pointing] • “Give me light” • Sit down • Flip a switch • Dialog box with buttons • Map with lamp indicators • “Turn on the room lights” User study preliminary conclusions • Users prefer speech interfaces • Gaze very helpful for interpreting specified device • Geometric knowledge needed for speech understanding

26. Demo Applications • Room Controller • Web pages generated from XML service descriptions • Remote sessions • Media control automation • (video: 00:28-00:45) • Mouse anywhere • (video: 05:55-07:00)

27. Concluding Comments • Ubiquitous computing needs middleware • Addressing, message passing & resource mgmt. • Geometric knowledge essential for good UIs • World model abstractions simplify application building • Perception is (still!) hard