Enabling Natural Interaction

1. Enabling Natural Interaction Randall Davis, Howard Shrobe Aaron Adler, Christine Alvarado, Mark Foltz, Tracy Hammond, Mike Oltmans, Metin Sezgin,Olga Veselova MIT Artificial Intelligence Lab

2. Consider This Device...

3. Natural Interaction? • What if we had to deal with one another as we deal with our software?

4. Let’s Talk

5. Our Vision

6. Our Model

7. Rationale Capture • Rationale capture doesn’t happen. • Because it’s too much trouble • Rationale capture tools don’t work • Because they’re too much trouble • The key: it needs to be less trouble than it’s worth. • The key to less trouble: break down the barriers

8. Sketch Understanding

9. Why Sketching? • Informal • Succinct • Natural • High in information content • Preserve topological and spatial properties

10. Sketch Understanding • Requires intelligence • at the signal level: early processing • at the interpretation level: handling ambiguity • at the engineering level: intended behavior

11. Other Research • Additional domain: software diagrams • Multi-modal interaction • Describing and learning new icons • Generating efficient recognizers

12. Sketching Software

13. Sketching Software

14. Sketching Software

15. Sketching Software

16. Sketching Software //Source file: c:\\video\\Deck.java public class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** * @roseuid 3C21348C0257 */ public Deck() { } } //Source file: c:\\video\\Game.java public class Game { public Graphics theGraphics; public Deck theDeck; public Player thePlayer; /** * @roseuid 3C21348C02E3 */ public Game() { } } //Source file: c:\\video\\Dealable.java public interface Dealable { }

17. Multi-Modal Interaction “Three identical, equally spaced pendulums…” • Graphical vs verbal communication: What’s best said, what’s best sketched?

18. New Icons • Define AndGate • line L1 L2 L3 • arc A • semi-circle A1 • orientation(A1, 180) • vertical L3 • parallel L1 L2 • same-horiz-position L1 L2 • connected A.p1 L3.p1 • connected A.p2 L3.p2 • meets L1.p2 L3 • meets L2.p2 L3 • A shape description language • A drawing sequence description language

19. Sketch Understanding Architecture Mechanical Engineering devices simple compositions primitive objects geometry rectangles, curves

20. Sketch Understanding Architecture Mechanical Engineering devices simple compositions primitive objects springs, joints geometry rectangles, curves

21. Sketch Understanding Architecture Mechanical Engineering devices simple compositions ratchet, latch primitive objects springs, joints geometry rectangles, curves

22. Sketch Understanding Architecture Mechanical Engineering devices mechanisms simple compositions ratchet, latch primitive objects springs, joints geometry rectangles, curves

23. Sketch Understanding Architecture Mechanical Software Engineering Engineering devices mechanisms simple compositions ratchet, latch primitive objects springs, joints objects, methods geometry rectangles, curves

24. Sketch Understanding Architecture Mechanical Software Engineering Engineering devices mechanisms simple compositions ratchet, latch hierarchies primitive objects springs, joints objects, methods geometry rectangles, curves

25. Sketch Understanding Architecture Mechanical Software Engineering Engineering devices mechanisms modules simple compositions ratchet, latch hierarchies primitive objects springs, joints objects, methods geometry rectangles, curves

26. Design Rationale Capture • Make it less trouble than it’s worth • Break down the barriers: eliminate interfaces • Natural interaction is enabled by intelligence • About sketching, gesturing • About the domain • About the task