Tae Soo Yun Dept. of Digital Contents Dongseo University Fall 2002 based on notes from

1. Advanced Topics in Virtual Reality Tae Soo Yun Dept. of Digital Contents Dongseo University Fall 2002 based on notes from Soon Ki Jung, KNU Wohn, KAIST ……

2. Table of Contents • Introduction : What is VR ? • Psychological and Cognitive Issues • VR System Anatomy • Virtual Perception • Interaction • Virtual Worlds: Representation, Creation and Simulation • Virtual Worlds: Rendering • Networked VR Systems and Shared Virtual Environements • Image-based Virtual Reality • Augmented Reality

3. Conceptual Model of VR Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor P-effectorv L-effector virtual object sensing avatar action virtual agent V-effector L-sensor P-sensor Logical devices for displacements, angles, events.

4. Chapter 3. VR System Anatomy 3-1. System Functional Overview 3-2. System-level Metaphors 3-3. Physical Sensors 3-4. Physical Effector: Visual Display 3-5. Physical Effector: Auditory Display 3-6. Physical Effectors: Haptic and Motion Display 3-7. Physical Effectors: Olfactory Display 3-8. Computing System 3-9. Case Study: Existing VR Systems

5. 3-1. System Functional Overview Topics to be discussed • Typical setup for a VR system • Conceptual model • Functional model -- a schematic diagram • Functional modules

6. A typical set-up

7. Conceptual Model of VR Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor P-effectorv L-effector virtual object sensing avatar action virtual agent V-effector L-sensor P-sensor Logical devices for displacements, angles, events.

8. Functional diagram rendering (Chap. 7) displaying (Sec. 3-4,5,6,7) simulation (Sec. 6-3,4) VW DB (Sec. 6-1) Virtual perception (Chap. 4) interaction (Chap. 5) Sensing (Sec. 3-3) VW Authoring (Sec. 6-2) (Sec. 3-8)

9. modules • sensing • virtual perception • interaction • simulation • rendering • displaying • virtual world DB

10. sensing and virtual perception • sensing • handles the physical sensors (and its controller) of the conceptual model • virtual perception • handles the transition from the physical sensors to the logical sensors.

11. Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor P-effectorv L-effector virtual object sensing avatar action virtual agent V-effector L-sensor P-sensor Logical devices for displacements, angles, events.

12. Glove Controller Hand gesture recognition Video camera Image processor Body gesture recognition Video camera Image processor Facial/face expression recognition Microphone Signal processor Speech recognition Sensing Virtual perception

13. Interaction rendering (Chap. 7) displaying (Sec. 3-4,5,6,7) simulation (Sec. 6-3,4) VW DB (Sec. 6-1) Virtual perception (Chap. 4) interaction (Chap. 5) Sensing (Sec. 3-3) VW Authoring (Sec. 6-2) (Sec. 3-8)

14. Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor P-effectorv L-effector virtual object sensing avatar action virtual agent V-effector L-sensor P-sensor Logical devices for displacements, angles, events.

15. Avatar의 정의, 모델링. • geometrical model • interaction model • interaction가능한 body parts? • avatar는 가상환경과 어떤 interaction을 할 수 있는가? • Logical sensor의 정의. • Logical sensor로부터 avatar(virtual effector)로의 mapping. • Avatar(virtual sensor)로부터 logical effector로의 mapping. • Run-time시 실제 일어나는 interaction결정.

16. Simulation rendering (Chap. 7) displaying (Sec. 3-4,5,6,7) simulation (Sec. 6-3,4) VW DB (Sec. 6-1) Virtual perception (Chap. 4) interaction (Chap. 5) Sensing (Sec. 3-3) VW Authoring (Sec. 6-2) (Sec. 3-8)

17. Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor P-effectorv L-effector virtual object sensing avatar action virtual agent V-effector L-sensor P-sensor Logical devices for displacements, angles, events.

18. Interaction에 따라 virtual world를 변화시킴 • Virtual world를 운영 • Kinematic simulation • dynamic simulation • behavioral simulation • behavioral simulation of autonomous agents • cognitive, emotional, physical

19. Rendering rendering (Chap. 7) displaying (Sec. 3-4,5,6,7) simulation (Sec. 6-3,4) VW DB (Sec. 6-1) Virtual perception (Chap. 4) interaction (Chap. 5) Sensing (Sec. 3-3) VW Authoring (Sec. 6-2) (Sec. 3-8)

20. Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor P-effectorv L-effector virtual object sensing avatar action virtual agent V-effector L-sensor P-sensor Logical devices for displacements, angles, events.

21. Simulation된 결과를 rendering. • Perceptualization of virtual worlds • image rendering • sound rendering • haptic rendering • others

22. Displaying rendering (Chap. 7) displaying (Sec. 3-4,5,6,7) simulation (Sec. 6-3,4) VW DB (Sec. 6-1) Virtual perception (Chap. 4) interaction (Chap. 5) Sensing (Sec. 3-3) VW Authoring (Sec. 6-2) (Sec. 3-8)

23. Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor P-effectorv L-effector virtual object sensing avatar action virtual agent V-effector L-sensor P-sensor Logical devices for displacements, angles, events.

24. Stereo image display Stereo image generation Headphone 3-D sound generation Haptic display Haptic signal generation

25. Virtual World DB rendering (Chap. 7) displaying (Sec. 3-4,5,6,7) simulation (Sec. 6-3,4) VW DB (Sec. 6-1) Virtual perception (Chap. 4) interaction (Chap. 5) Sensing (Sec. 3-3) VW Authoring (Sec. 6-2) (Sec. 3-8)

26. Representation of virtual worlds • “scene graph” • Big issue • expressiveness vs. efficiency

27. 3-2. System-level Metaphors • How is a virtual world presented to the user? 1. On-the-table (desktop, fish tank, work bench,…) 2. Through-the-window (inside-the-vehicle, …) 3. Immersive • encumbered (body suit, HMD,…) • CAVE, Dome • Neural-connection 4. Third person (mirror, reflexive)

28. Virtual workbench

30. Through-the-window

31. CAVE “A Review of Tele-Immersive Applications in the CAVE Research Network”, J. Leigh, et.al., Proc. IEEE Virtual Reality ’99, pp. 180-187, March 1999.

33. Third person (reflexive, mirror) Camera Avatar VR system Large Display Participant

34. Virtual Stage

35. Chroma-keying 3rd Person VR (2D) developments 3rd Person VR (3D) Virtual Studio

36. Advantages • full-body interaction • unencumbered interface • easily shared virtual experience • Disadvantages • slow/coarse/poor interactions • (early systems) 2D VE, interactions • must watch the VE face to face. • Applications • entertainment, education • broadcasting, film making

37. 3-3. Physical Sensors Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor P-effectorv L-effector virtual object sensing avatar action virtual agent V-effector L-sensor P-sensor Logical devices for displacements, angles, events.

38. 1. Conventional 2D Interfaces 2. Position trackers 3. Gloves 4. Others

39. Body parts to be tracked hand, finger eye head whole body Performance specifications static accuracy dynamic accuracy latency update rate signal to noise ratio registration Position trackers

40. Tracking methods • mechanical • magnetic • optical • stereo, multiple cameras • time-of-flight, phase • structured light • acoustic • inertial • GPS • Research issues • tracking and prediction • sensor fusion

41. Gloves • Finger flexions : • Optical fiber sensors [VPL dataglove] • Hall Effect Sensor [Exos Dexterous Hand Master] • Strain gauges based on flexible plastic with a constant resistive surface [Mattel’s Power Glove] • Resistive bend-sensing technology [Virtex Cyberglove] • Orientation / position • Electromagnetic tracking system [Polhemus Isotrak] • Ultrasonic transducer / receiver [Mattel’s Power Glove] • Electromechanical, acoustic, optical tech.. [VPL dataglove] [Power Glove]

42. Others • 3D mouse • wand • treadmill • bicycle, wheel chair, shopping cart • Neural interface • EMG (electro-myograph) -- muscle • EEG (electro-encephalograph) -- brain • EOG (electro-oculograph) -- eye

43. 3-4,5,6,7. Physical Effectors Virtual environment Human H-sensor perception cognition motion control H-effector V-sensor P-effectorv L-effector virtual object sensing avatar action virtual agent V-effector L-sensor P-sensor Logical devices for displacements, angles, events.

44. Physical effectors • visual display • auditory display • haptic display • motion display • olfactory display

45. 3-4. Physical Effector: Visual Display • “ideal” display to match 20/20 vision • resolution: 5,000 x 5,000 • fov: 90 degrees • update rate: 40 ~ 50 fps

46. Objectives • To develop the hardware (visual devices) • To develop the algorithm for the hardware in such a way that the visual display match the characteristics of the human visual system. Research issues • methods for perfecting display devices (e.g., calibration,...) • psychological issues (e.g., sensory adaptation, depth perception, …) • human perception-based displaying techniques

47. Topics to be covered 1. Image generation: principle • device-independent issues 2. Image generation: technology • various types of displays 3. Head mounted display

48. 3-5. Physical Effector: Auditory Display • Research issues • Principle of 3-D sound generation • 3-D sound generation techniques • Psychological effects of altered auditory environments

49. 3-6. Physical Effector:Haptic and Motion Displays • Force, pressure, tactile feedbacks • Haptic interface requires sensing capability. • Research issues • Haptic science (studies on human haptics) • Tool-hand system • Creating the haptic illusion • Interaction interface of haptic and vision • Texture, temperature devices

50. 1. tactile display pin matrix