Distributed Sensing Techniques for Ubiquitous Mobile Devices

1. Ken Hinckley kenh@microsoft.com Microsoft Research Nov. 20th, 2003 Distributed Sensing Techniques for Ubiquitous Mobile Devices

2. Who am I anyway? • Research Scientist, PhD in Comp Sci • Background in Human-Computer Interaction • Interests • Input Devices • Interaction Techniques • Sensors • Mobile Devices • Ubiquitous Computing

3. Some Recent Projects • “Meat & Potatoes” • Mouse Scrolling wheel – accelerated scrolling

4. Accelerated Scrolling Wheel • How do users roll the wheel? • (1) trying to get somewhere fast, or • (2) reading • Roll faster  move further • No loss of control • The user does nothave to learn anything new • Exp. study: Does no harm, users can scroll faster

5. Some Recent Projects • “Meat & Potatoes” • Mouse Scrolling wheel – accelerated scrolling • Office Keyboard

6. MS Office KeyboardHugh McLoone, Ken Hinckley, Ken Fry, Steve Bathiche • Navigation controls on left • Scrolling - wide wheel • Web - Forward / Back] • Windows - AppToggle • Cut, Copy, Paste keys on left • RH: Compound selection / IP LH: articulation of command

7. Uncanny Coincidence #1 • October 2003 •  Office Keyboard Winner of HFES 2003 User Centered Consumer Product Design Award! •  Office Keyboard discontinued by Microsoft

8. Some Recent Projects • “Meat & Potatoes” • Mouse Scrolling wheel – accelerated scrolling • Office Keyboard – bimanual navigation controls • More far-ranging research • Sensing Techniques for Mobile Interaction

9. Sensing Techniques for Mobile Interaction • Sensors: • Tilt – 2 axis accelerometer • Touch • Proximity • inexpensive, low power • What new features might inexpensive, ubiquitous sensors enable? • VIDEO • ACM UIST 2000 Best Paper Award!

10. Uncanny Coincidence #2 • 11/20/2003 – Ken advocates coming ubiquity of tilt & other sensors sensors • 11/18/2003 – Conveniently to make Ken’s point, Toshiba announces Portege M200 Tablet PC • Tilt fwd/back to scroll • Tilt L/R to flip pages • Orient to change Portrait / Landscape

11. Some Recent Projects • “Meat & Potatoes” • Mouse Scrolling wheel – accelerated scrolling • Office Keyboard – bimanual navigation controls • More far-ranging research • Sensing Techniques for Mobile Interaction • Distributed Sensing Techniques for Ubiquitous Mobile Devices

12. Wireless Networks • Still an immature technology – • Slow, flaky • Do people really want to read web pages on their cell phones? • Sensor nets - applications for mobile devices? • Odd phrase: not what it is, but what it lacks • In 100 years Wireless Network = ???

13. Horseless Carriage?

14. 2003: Isolated Digital Islands

15. Wireless Fills in Gaps between Devices: But Which Ones?

16. Users need techniques to intuitively form bridges between devices How do users name the devices to connect? What is type / purpose of the connection? Parameters? (Who copies what, to where?) Name That Device

17. User activity that is sensed by two or more devices in a distributed system Shared with nearby participating devices via wireless networking System synchronizes time & puts the pieces back together Our Strategy:Synchronous Gestures

18. Shaking Hands

19. Bowing

20. Making a Toast

21. VIDEO– Perceptive TabletPC’s

22. Any “bumps” while devices not being held are ignored Example “Bump” Data

23. Bumping does more than name the devices: Edges to join: Left, Right, Top, Bottom 2-axis tilt sensor gives this information Direction (hierarchy in connection) One tablet is base, one tablet is connecting Without this, unclear which tablet is “boss” Touch sensor: which device is held? Bumping has Implicit Parameters

24. How Synchronous Gestures Enable Wireless Matchmaking

25. Server may be hosted in environment,or on a tablet. Participating Devices Connect to Server Sync Gesture Server

26. Send Sensor Data to Server Sync Gesture Server

27. Time Sync Sync Gesture Server

28. Bump Recognized! Sync Gesture Server

29. Alternatively can have “private” connection via Sync Gesture Server Server Gives Partners Each Other’s IP Address Sync Gesture Server

30. Connect to Peer & Link Displays Sync Gesture Server

31. Can’t synchronize with everyone: n(n-1)/2 limited resources / power false positives through sheer chance Ideally, should sync only w/ co-located devices Possible scoping mechanisms peer-to-peer: anyone in radio range sync’s proximity server: discover nearby devices through 802.11 signal strengths (John Krumm) Open Problem:Who to Synchronize With?

32. Wireless signal strength provides proximity Bumping offers explicit step of intentionality that user has control over selection done in real world via manipulation Bumping provides additional information edges to join, type of connection, direction, etc. Proximity provides list of devices to synchronize with Why Not Just Use Proximity Sensing?

33. Other Ways to Use Bumping • Sharing information –mutual or one way • Multiple tablets • Face-to-face collaboration • VIDEO #2

34. Bumping works well, intuitive, fun But requires hardware that is not (yet ) in common use A few operations possible, but seems hard to add further variations or semantics Started thinking about generalizing the issues we encountered The result was… What Next?

35. Ken Hinckley Gonzalo Ramos (Intern) Francois Guimbretiere (Visiting Researcher) Patrick Baudisch Marc Smith Work currently submitted to CHI 2004 Stitching: Pen Gestures that Span Multiple Displays

36. Move the pen Cross over bezel Finish stroke on nearby tablet System infers connection Yields versatile framework for connecting pen-operated devices Stitching

37. VIDEO

38. How is a connection established? What type of connection is required? What information is shared? What is the spatial relationship btw devices? How do stitching gestures coexist with other uses for the pen? How do users share physical space? Implemented a photo sharing app Central Design Questions

39. There is nothing special about the pen! Recognizes timing & dynamics of pen trajectory Pen exits from edge of one screen & enters at edge of another within a timeout 1. Establishing a Connection Tablet 2 Tablet 1 Δt

40. User selects a multi-device command e.g., Pie menu at end of stitching gesture 2. Type of Connection

41. Many commands may need operands Lasso to select, continue to other display to stitch. 3. What Information to Share?

42. Fits a line to absolute coords of user’s pen stroke 4. What is Spatial Relationship: Automatic Screen Calibration

43. Stitching can occur in Tracking or Dragging Tracking is separate layer on top of GUI Dragging is only option for PDA’s Must undo drags that turn out to be stitches 5. Coexistence of Stitching with Traditional Pen Use

44. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 6. How Do Users Share Physical Space?

45. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 6. How Do Users Share Physical Space? • Homework assignment: • Sit right next to someone at airport (when it is not necessary to do so) • Time how long it is before the other person leaves

46. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 6. How Do Users Share Physical Space?

48. Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Proxemic Lessons for Stitching

49. Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Establish and Relax Users want to Establish a connection, but then Relax increasing social tension by quickly exiting Observed lean back, short stitches; transporter popular One argument against remote menus Flexibility: Allow users to be involved or not Proxemic Lessons for Stitching

50. Intimate 0-18” Personal 18- 30” Social: 4’ – 7’ Public 12’ or more Distances vary by culture Touching is taboo in non-contact cultures E. T. Hall’s “Distances in Man” } Within arm’s reach } Beyond arm’s reach