1 / 25

Building the Design Studio of the Future

Building the Design Studio of the Future. Aaron Adler Jacob Eisenstein Michael Oltmans Lisa Guttentag Randall Davis October 23, 2004. Sketching Is Not the Whole Picture. Speech and Gestures. Express more precise spatial and structural relationships

lukas
Télécharger la présentation

Building the Design Studio of the Future

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Building the Design Studio of the Future Aaron Adler Jacob Eisenstein Michael Oltmans Lisa Guttentag Randall Davis October 23, 2004

  2. Sketching Is Not the Whole Picture Building the Design Studio of the Future

  3. Speech and Gestures • Express more precise spatial and structural relationships • Relate sketches that are drawn from different perspectives • Describe temporal events or paths of motion • Disambiguate using multiple modalities Building the Design Studio of the Future

  4. Roadmap Multimodal Communication Multimodal Understanding Recognition Components Multimodal Intelligent Design Studio Exploratory Studies Human Dialogue Dialogue Management - Out-of-vocabulary terms WoZ Dialogue Studies Usability - Possible design mistakes - Illuminate unclear parts Domain Modeling Domain Reasoning Building the Design Studio of the Future

  5. Outline • Sketching is not the whole picture • Roadmap • Trimodal Studies • Preliminary Components • Speech and Sketching • Gesture and Speech Building the Design Studio of the Future

  6. Outline • Sketching is not the whole picture • Roadmap • Trimodal Studies • Preliminary Components • Speech and Sketching • Gesture and Speech Building the Design Studio of the Future

  7. Interesting Questions to Study • Sketch • What information is best communicated with sketching? • How do people sketch when they’re explaining things? • How do they sketch when they’re taking notes? Building the Design Studio of the Future

  8. Interesting Questions to Study • Speech • What information is best communicated with speech? • How do people talk when they’re drawing? How can ASR better support speech in the context of other modalities? • How should computers generate and understand back-channel feedback? Building the Design Studio of the Future

  9. Interesting Questions to Study • Gesture • What information is best communicated with gesture? • Which features of gesture are idiosyncratic? Which are universal? • What does gesture tell us about discourse structure? • Are gestures cues for disfluencies? Sentence breaks? • How can we build embodied agents with realistic body language? Building the Design Studio of the Future

  10. Trimodal Interaction Study • How are speech, gesture, and sketching used together? • 15 pairs of participants • Two video cameras, a Mimio, and a Tablet PC • 4.5 hours of data Building the Design Studio of the Future

  11. Looking Ahead to Recognition • Captured digital video • Colored gloves used for future hand tracking • Headset microphones • Time-stamped sketch data Building the Design Studio of the Future

  12. Diagrams and objects Building the Design Studio of the Future

  13. Trimodal Setup Speaker Listener Building the Design Studio of the Future

  14. Listener Setup Speaker Setup Trimodal Setup Building the Design Studio of the Future

  15. Details to Observe • Structure then function • Back channel communication • Collaboration between speaker and listener Building the Design Studio of the Future

  16. Movie Clip I Building the Design Studio of the Future

  17. Movie Clip II Building the Design Studio of the Future

  18. Movie Clip III Building the Design Studio of the Future

  19. Early Results • We identified two phases in the explanations: • Structure phase • Function phase • Participants used back channel communication during the explanations • The amount of sketching the listener did varied Building the Design Studio of the Future

  20. Outline • Sketching is not the whole picture • Roadmap • Trimodal Studies • Preliminary Components • Speech and Sketching • Gesture and Speech Building the Design Studio of the Future

  21. Results: Speech and Sketching • Filled pauses important: “ahh”, “umm” • Key phrases: “there are”, “and” Building the Design Studio of the Future

  22. Multimodal Understanding and Integration • Combining speech and sketching we can make it easier to perform some operations “There are three identical pendulums.” “There are three touching pendulums.” Building the Design Studio of the Future

  23. so this piece goes here and this… deictic tracing trajectory Speech-Gesture Integration Time • Align referential pronouns and gestures to the diagram “ this piece is made of wood” • “it moves back and forth like this” Building the Design Studio of the Future

  24. Results: Speech and Gesture • Corpus-based approach • Feature set based on linguistic knowledge • Improves performance • 84% F-measure for baseline system • 95% F-measure for optimization system Building the Design Studio of the Future

  25. Conclusion • Intelligent Design Studio of the Future • Trimodal Corpus • Empirical Results • Initial Components Building the Design Studio of the Future

More Related