1. Graphical User Interfaces inVirtual Reality Patrick Hartling Virtual Reality Applications Center IEEE VR 2002

2. Overview • Motivation • Issues to consider • Previous work • Current user interface work at VRAC

3. Motivation • Why have GUIs in virtual reality? • GUIs are everywhere in the computing world • Most computer users understand how to use a GUI • Why not have GUIs in VR?

4. Issues with GUIs inVirtual Reality • Positives • Typically more flexible and more extensible than other input devices • No overloading of button presses, gestures, etc. • More portable between VR systems than hardware input devices • Familiarity from desktop experience

5. Issues with GUIs inVirtual Reality (cont’d.) • Negatives • No direct physical correspondence • In VR, a user could grab an object with his/her hand or use a GUI • No GUI in real world for moving the object • Remote users may not see GUI [Park00] • Use of the interface may be difficult • Collision detection • Placement • Immersive interfaces have to re-invent GUI technology

6. Previous Work • Virtual User Interface (VUI) • Originally developed at Iowa State University [Heath98] • 3D immersive user interface • May be an open source release from Fuel Tech

7. Previous Work • Java Interface to Virtual Environment (JAIVE) • Used Java interface on handheld computer [Hill00] • Communicated with C++ applications • Interface could be extended from C++ using JAIVE protocol

8. Previous Work (cont’d.) • 3DI Group • Doug Bowman, Virginia Tech • Investigating differences in system interaction techniques between VE displays • Active on 3DUI mailing list

9. Tweek • Features • GUI panels are dynamically loaded components • Panels use standard 2D GUI “widgets” • Dynamic interface update and extension • Multi-language capabilities

10. Use of 2D Interface • Reasoning • Some interaction methods are best suited to 2D interfaces • Database entry, annotations, precision input • 2D interface technology is very mature and familiar

11. Hybrid Approach • Reuse of GUI • Desktop simulation • PDA in a projection-based VR system • Immersive 3D space

12. Same GUI, Multiple Uses • PDA in Projection System • Offers force feedback • High-resolution display • Back-lit display is needed • Widely available • A person with a PDA can walk into the VR system and get the GUI dynamically

13. Same GUI, Multiple Uses (cont’.d) • Desktop simulation • Standard GUI interface used to control VR software in simulator mode • Immersive 3D space • Incorporates 2D GUI into 3D visualization • 3Dwm (www.3dwm.org) • VR VNC viewers • Part of original design, still a future goal

14. Tweek Design • Language tools • C++ VR applications • Java-based GUI composed of JavaBeans • Beans could be “pushed” to the GUI while the user is in the virtual space • Dynamic extension of interface • CORBA communication between C++ and Java

15. Example Tweek Interface • Navigation tool

16. Future Work • Full implementation of JavaBean “pushing” to Tweek GUI • Use of 3Dwm or other tool for bringing 2D GUI into 3D space

17. Online Information • 3DUI • www.mic.atr.co.jp/~poup/3dui.html • Tweek • www.vrjuggler.org • Under “Projects” block • Tweek  Documentation • Contains guides and references

18. Conclusion • Questions?

19. References • [Park00] Park, K. et al. “Lessons Learned from Employing Multiple Perspectives in a Collaborative Virtual Environment for Visualizing Scientific Data”. In Proceedings of ACM Conference on Collaborative Virtual Environments, San Francisco, California, September 10–12, 2000. • [Heath98] Heath, D. “Virtual User Interface (VUI): A Windowing System for VR”. In Proceedings of 2nd Immersive Projection Technology Workshop, Ames, Iowa, May 11–12, 1998. • [Hill00] Hill, L., Usability of 2D Palmtop Interaction Device in Immersive Virtual Environment. Master’s thesis, Iowa State University, Ames, IA, 2000. • [Watsen99] Watsen, K. “A Handheld Computer as an Interaction Device to a Virtual Environment”, In Proceedings of 3rd Immersive Projection Technology Workshop, Stuttgart, Germany, May 10–11, 1999.