Enabling Natural Interaction

1. Enabling Natural Interaction Randall Davis Aaron Adler, Sonya Cates, Jacob Eisenstein, Tracy Hammond, Mike Oltmans, Metin Sezgin, Chen Li, David Pitman MIT CSAIL

2. Consider This Device...

3. Our Model • The designer sketches with pen and paper • The observer interprets the sketch • The observer and designer interact

4. Sketches are Ubiquitous

5. Sketches are Dead

6. Magic Paper

7. Demo

8. More Complex Example

9. More Complex Example

10. More Complex Example

11. A Prevasive Theme Natural interaction is knowledge-based.

12. Blackboard Hand-codedrecognizers Architecture Overview Early processing

13. Describing New Symbols • Define AndGate • line L1 L2 L3 • arc A • semi-circle A1 • orientation(A1, 180) • vertical L3 • parallel L1 L2 • same-horiz-position L1 L2 • connected A.p1 L3.p1 • connected A.p2 L3.p2 • meets L1.p2 L3 • meets L2.p2 L3

14. Learning New Symbols • From hand-drawn example to a shape description • Define AndGate • line L1 L2 L3 • arc A • semi-circle A1 • orientation(A1, 180) • vertical L3 • parallel L1 L2 • same-horiz-position L1 L2 • connected A.p1 L3.p1 • connected A.p2 L3.p2 • meets L1.p2 L3 • meets L2.p2 L3

15. Learning New Symbols • Knowledge about human perception: Some properties are more important • Experiments by Goldmeier (1936) identify perceptually relevant properties and relations; singularities

16. Blackboard Hand-codedrecognizers Symbolic Descr. New recognizers Learn icon Code Gen. recognizer Multi-modal Architecture Overview Early processing Shape & Seq. Description 1 example

17. Descriptions of Objects Application Pixels Objects Sketch-Enabled Interaction Architecture Overview Appl’n 2 Appl’n 3

18. Multi-Modal Interaction

19. Multi-Modal Interaction “Three identical touching pendulums…” • Graphical vs verbal communication: What’s best said, what’s best sketched?

21. Understanding Gestures • Lexicon: What gestures do people make? • Depends on task. • Our task: explaining how something works.

22. Understanding Gestures • 96% of gestures refer to the diagram. • Two-handed gestures are common. • Deixis is more frequent.

23. Sketching and Talking:Laboratory Software

24. Natural, Unrestricted Input • Domain: Sketching family trees • Lexicon male female child-of married-to divorce

25. 1 2 10 7 8 9 11 12 5 6 4 3 Natural, Unrestricted Input • Task is incremental • Signal is noisy • Styles vary • Segmentation is difficult • The signal is 2-d, non-chronological

26. Low Level Context Use

27. Higher Level Context Lexicon male female child-of married-to divorce • Grammar

28. Higher Level Context

29. Description Matching is Slow

30. Find Possible Objects Quickly

31. Find Possible Objects Quickly

32. Domains • Mechanical engineering • Family trees • Software design • Circuit analysis • Medical applications

33. Sketching Software

34. Sketching Software

35. Sketching Software

36. Sketching Software //Source file: c:\\video\\Deck.java public class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** * @roseuid 3C21348C0257 */ public Deck() { } }

37. Sketching Software //Source file: c:\\video\\Deck.java public class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** * @roseuid 3C21348C0257 */ public Deck() { } }

38. Circuit Analysis

39. Specifying Shapes

40. Hardware

41. Summary • Once the sketch is understood, many things are possible. • Break down the barriers: eliminate interfaces • Natural interaction is enabled by intelligence • About sketching, gesturing, speech • About the domain • About the task