1. Institut National de Recherche en Informatique et Automatique Stanislaw Borkowski thesis director: James L. Crowley Jury: Steerable Interfaces for Interactive Environments Andreas Butz (UM), Joëlle Coutaz (UJF), Alex Pentland (MIT), Pierre Wellner (IDIAP) INRIA Rhône-Alpes June 26, 2006

2. What is a user interface? • User Interface (UI): aggregate of physical entities or information bound to these entities A

3. Mobile UI’s • Steerable UI: • can be relocated in space • position is mediated by the computer system • Portable UI: • can be relocated • position is directly controlled through physical contact Mobile UIs Steerable UIs Portable UIs A

4. Mobility in current IT • Steerable interfaces • Conventional GUI (steerable output) • X11 session teleporting [Richardson93] • Portable interfaces • Wearable computers • Cell phones • Personal Digital Assistants • Laptops • ….

5. Mobility in ambient computing • Multiple displays embedded in the environment • Large size displays • Mobile interaction resources, both portable and steerable [Arias00] [Streitz99] [Pinhanez01]

6. Why steerable? • Flexibility in resources usage • New forms of Human-computer interaction • New forms of Human-Human interaction

7. Current situation – summary Problem: • Need for steerable UIs • No predictive models Solution: • Provide enabling technology • Explore interaction techniques • Evaluate the value of steerable UIs

8. Outline • Mobility in IT • Steerable UIs • Mobile projected UI • Mobile UIs for collaborative work • Conclusions

9. State of the art • EasyLiving [Brumitt00] • Tic-Tac-Toe [Pinhanez05]

10. State of the art – limitations • UI is observable at standstill • Limited spatial controllability • Only predefined locations • Planar surfaces only Requirements for steerable UIs: • Continuous observability and controllability

11. Outline • Mobility in IT • Steerable UIs • Mobile projected UI • Prototype implementation[in collaboration with J. Letessier] • Evaluation – latency estimation • Mobile UIs for collaborative work • Conclusions

12. The Steerable Camera Projector(2002) Other steerable projection systems: • The Everywhere Display (IBM 2000) • Fluid Beam (Fluidum consortium 2002) • SCP from Karlsruhe (UKA 2004)

13. Steerable display (2002)

14. User-centric approach End-users: • Latency limits < 50ms • Easy setup, no maintenance • Reliability / predictability Developers: • Abstraction: be relevant • Isolation: allow integration • Contract: offer quality of service

15. Pragmatic approach • Black-box services • BIP(Basic Interconnection Protocol) • BIP implementation ≈ SOA middleware • service/service and service/application communication • service discovery (standards-based)

16. Interactive system SCP software Human and Environment Interaction events Application Display orders

17. Human and Environment Interactive system Interaction detector Frame grabber Application SCPdisplay SCPcalibrator SCPcontroller A

18. Human and Environment Interactive system Interaction detector Frame grabber Application SCPdisplay SCPcalibrator SCPcontroller

19. Screen Video projector Source image Projection on arbitrary oriented planar surfaces Light source User’s perception

20. Screen Video projector Image to project Source image Projection on arbitrary oriented planar surfaces Light source SCPdisplay User’s perception

21. Projection on arbitrary oriented planar surfaces User’s view Image to project

22. Human and Environment Interactive system Interaction detector Frame grabber Application SCPdisplay SCPcalibrator SCPcontroller

23. 3 Y 2 1 Q Sensor-centric environment model 2 3 1

24. Display surface detection Screen

25. The Portable Display Surface

26. Human and Environment Interactive system Interaction detector Frame grabber Application SCPdisplay SCPcalibrator SCPcontroller

27. Interactive widgets projected on a portable display surface

28. Luminance-based button widget

29. Touch detection • Locate widget in the camera image • Estimate occlusion • Update widget state

30. Robustness to clutter

31. Assembling occlusion detectors

32. Striplet – the occlusion detector x y

33. Striplet-based SPOD SPOD – Simple-Pattern Occlusion Detector

34. Striplet-based button

35. SPOD-based calculator Accelerated video

36. Outline • Mobility in IT • Steerable UIs • Mobile projected UI • Prototype implementation • Evaluation – latency estimation • Mobile UIs for collaborative work • Conclusions

37. PCI A/D converter Frame Grabber Graphic Card OpenGl render Latency estimation CPU Imalab shell Image processing PDS

38. PCI A/D converter Frame Grabber Graphic Card OpenGl render Latency estimation Video sequence capture CPU Imalab shell Projection of the bar Image processing Plastic bar Fan Regulated power supply

39. PCI A/D converter PCI A/D converter Frame Grabber Frame Grabber ~17ms CPU Imalab shell Image processing ~70ms Graphic Card OpenGl render CPU Imalab shell ~32ms Graphic Card Latency estimation – results A + up to 51ms!!!

40. Human and Environment Interactive system Interaction detector Frame grabber Application SCPdisplay SCPcalibrator SCPcontroller

41. Outline • Mobility in IT • Steerable UIs • Mobile projected UIs • Mobile UIs for collaborative work • ContAct application • User study – comparison of different take-over techniques • Conclusions

42. ContAct – a system for authoring presentations Collaboration through interface mobility

43. ContAct application setup • Wide angle camera • Tabletop camera • Steerable Camera Projector • Portable Display Surface

44. ContAct application GUI

45. Outline • Mobility in IT • Steerable interface prototype • Mobile UIs for collaborative work • ContAct application • Taking control: a comparative user study [in collaboration with J. Maisonnasse and J. Letessier] • Conclusions

46. Evaluation of techniques for taking control Objectives: • Determine the preferred control taking technique • Evaluate the impact on the task completion performance • Evaluate user acceptance of steerable interfaces

47. Experimental setup Hardware: • Steerable Camera Projector • Microphone headsets • Portable Display Surface GUI Software: • Speech detector [D. Vaufreydaz] • Conversation modeling [J. Maisonnaisse] • Finger tracking [J. Letessier] • PDS tracking • Drawing application Users

48. The User Interface

49. The task Collaborative reconstruction of a graph

50. The task Collaborative reconstruction of a graph User 1 User 3 User 2