High Performance Pen Interfaces

1. Ken Hinckley Microsoft Research October 22nd, 2004 kenh@microsoft.com http://research.microsoft.com/~kenh High Performance Pen Interfaces

2. Pen Computing Now?

3. Project #1: Stitching • Ken Hinckley • Gonzalo Ramos • Francois Guimbretiere • Patrick Baudisch • Marc Smith

4. Wireless Networks • Still an immature technology – • Flaky, hard for users to configure, slow, … • Most research has focused on small-screen presentation of web pages • What are the completely new & unique affordances of wireless connectivity? • Odd phrase: not what it is, but what it lacks

5. Wireless Network 2004 = Horseless Carriage 1904? • Are Wireless Networks really just wired networks without the wires? • Or are they something completely different?

6. Wireless Device Soup: Which links does the user want?

7. Users need techniques to intuitively form bridges between devices How do users name the devices to connect? What is type / purpose of the connection? Parameters? (Who copies what, to where?) Name That Device

8. Pen stroke that spans displays Move the pen Cross over bezel Finish stroke on nearby tablet System infers connection Stitching

9. How is a connection established? What type of connection is required? What information is shared? What is the spatial relationship between devices? How do users share physical space? Explored in context of photo sharing app Central Design Questions

10. AVI’04 ACM Advanced Visual Interfaces 2004 VIDEO

11. Recognizes timing & dynamics of pen trajectory There is nothing special about the pen! Wireless signal strength determines nearby devices Tablet 2 Tablet 1 Δt 1. Establishing a Connection

12. User selects a multi-device command e.g., Pie menu at end of stitching gesture 2. Type of Connection

13. Many commands may need operands Lasso to select, continue to other display to stitch. 3. What Information to Share?

14. Fits a line to absolute coords of user’s pen stroke 4. What is Spatial Relationship: Automatic Screen Calibration

15. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 5. How Do Users Share Physical Space?

16. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 6. How Do Users Share Physical Space? • Homework assignment: • Sit right next to someone at airport (when it is not necessary to do so) • Time how long it is before the other person leaves

17. SociologyProxemics: How people use space Invisible bubble that surrounds an individual 5. How Do Users Share Physical Space?

20. Don’t require contact : touching is taboo “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Proxemic Lessons for Stitching

21. Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Gives users flexibility to be involved, or not: mood, task, … Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Proxemic Lessons for Stitching

22. Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Gives users flexibility to be involved, or not Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Support varying relative body orientation Proxemic Lessons for Stitching

23. 6 conversations 3 1 0 Proxemics: Orientation of Spaces • Communication patterns change wrt orientation • Stitching supports: • Shoulder-to-shoulder • Face-to-face • At 90 angles Relative frequency:

24. Multiple Devices: Cooperative Stitching User 1 User 2 User 4 User 3 Ongoing work

25. Multiple Devices: Cooperative Stitching User 1 User 2 User 4 User 3 Ongoing work

26. Project #2: Scriboli • Ken Hinckley • Patrick Baudisch • Gonzalo Ramos • Francois Guimbretiere

27. Round trip problem on Tablet PC • 4 trips (+ some xtra) for copy/paste

28. Round trip problem • 4 trips (+ some xtra) for copy/paste

29. Round trip problem • 4 trips (+ some xtra) for copy/paste

30. Round trip problem • 4 trips (+ some xtra) for copy/paste

31. Round trip problem • 4 trips (+ some xtra) for copy/paste

32. Round trip problem • 4 trips (+ some xtra) for copy/paste

33. Round trip problem • 4 trips (+ some xtra) for copy/paste

34. Round trip problem • 4 trips (+ some xtra) for copy/paste

35. Round trip problem • No right click, kbd shortcuts to avoid this!

36. GEdit – Kurtenbach & Buxton Delete one obj. Copy – add final “C” to move Delete group – end stroke inside lasso • One of the first works considering pen phrases • Every stroke is gesture • Only 3 cmds • Group vs. single obj. inconsistent Move – end stroke outside. Objects move on PenUp.

37. SKETCH – Brown University • Adding 1 more gesture breaks everything • Specific to 3D, difficult to learn how to use

38. Marking Menus • Prompted BlindScale independence • Self-revealing: prompts w/ available cmds • Rapid expert use: blind marking • Menu appears only after 1/3 second delay • Size of mark does not matter

39. FlowMenu – Francois Guimbretiere • Integrates command selection with continuous parameter manipulation • But no “blind marking,” size of gesture matters Crossing inout chooses Item… submenu Crossing outin chooses Move command Dragging continues

40. Tivoli • Tivoli project @ Xerox PARC • We wanted quick, informal feel of scribbling

41. Learning Cursive • building blocks of writing as basis for pen UI? • not natural handwriting recognition

42. The Deep Structure of Handwriting • The “scribble” of Scriboli • Phasic muscle contractions: • fast, brief, usually repetitive • Pre-programmed, internally driven • “The conversion of those […] O’s to real words and sentences happens when small variations in the amount of verticality or laterality […] are introduced into the movement. The extreme orderliness and predictability of individual handwriting is due to this mode of generating and controlling the movement.”F. R. Wilson, The Hand, p. 163

43. Design Philosophy of Scriboli • RAPID • Fast motions, repeatable, no waiting / dwelling, minimal demands on visual attention • UNAMBIGUOUS • No guessing games • No reco unless I tell the system otherwise • (There is almost no reco in Scriboli) • EXPRESSIVE • Supports many commands, easy to grow design • Mechanisms for a wide span of pen apps, not restricted to specific niche application

44. Punct. space capitals period comma object verb indirect obj. Written text separates words sentence start sentence end delimits clause “The cat sat on the mat” Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase Grammar for pen input • Scriboli proposes fundamental building blocks • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases

45. Ink vs. Gesture • Who does the recognition? • Ink: strokes intended primarily for human • Gesture: strokes intended for computer • Virtually all pen systems make this distinction at some level • Very seductive to attempt to do automatically • Many systems spend most or all effort just on this • Any pen gesture might occur in freeform input • Scriboli uses explicit ink/gesture mode switch

46. Ink/Gesture study w/ U. Washington • 5 Techniques: • Barrel Button • Tap+hold • NPH button • Pen Pressure • Pen Eraser

47. Punct. space capitals period comma object verb indirect obj. Written text separates words sentence start sentence end delimits clause “The cat sat on the mat” Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase Grammar for pen input • Scriboli proposes fundamental building blocks • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases

48. Delimiters: Single-stroke lasso selection + marking • Initial part of stroke is the scope • Pigtail is the delimiter • Tail interpreted as the mark

49. What is a Delimiter? • A delimiter is “something different” about the pen input stream • Determines lexical structure of pen stroke • Pigtail only intended as delimiter during Gesture mode • Not an automatically inferred mode switch

50. Punct. space capitals period comma object verb indirect obj. Written text separates words sentence start sentence end delimits clause “The cat sat on the mat” Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope(lasso,line, …) Marking menu Crossing manip. phase Grammar for pen input • Scriboli proposes fundamental building blocks • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases