Virtual Reality

1. Virtual Reality Dr. Yan Liu Department of Biomedical, Industrial and Human Factors Engineering Wright State University

2. Introduction • What is Virtual Reality (VR) or Virtual Environment (VE) • A medium composed of interactive computer simulations that sense the participant’s position and actions and replace or augment the feedback to one or more sense, giving the feeling of being mentally immersed or present in the simulation (a virtual world) (Sherman & Craig, 2003) • Four Key Elements in Experiencing VR (Sherman & Craig, 2003) • Virtual world • An imaginary space, often (but not necessarily) manifested through a medium • Immersion • Having a sense of presence within an environment; this can be purely a mental state, or can be accomplished through physical means • Mental immersion • A state of being deeply engaged, with a suspension of disbelief • Physical immersion • Bodily entering a medium

3. Introduction (Cont.) • Four Key Elements in Experiencing VR (Sherman & Craig, 2003) • Sensory feedback • Visual/aural/haptic feedback to participants, based on some aspects of their physical positions • Interactivity • In a virtual reality experience, participants are able to move around and change their viewpoint, generally through movements of their head • Four Technologies that are Crucial for VR (Brooks, 1999) • The visual (possibly also aural and haptic) displays that immerse the user in the virtual world and that block out contradictory sensory impressions from the real world • The graphics rendering system that generates, at 20 - 30 frames/second, the ever-changing images • The tracking system that continually reports the position and orientation of the user’s head and limbs • The database construction and maintenance system for building and maintaining detailed and realistic models of the virtual world

4. Introduction (Cont.) • Four Technologies that are Important for VR (Brooks, 1999) • Synthesized sound, displayed to the ears, including directional sound and simulated sound fields • Directional sound is a technology that concentrates acoustic energy into a narrow beam so that it can be projected to a discrete area, much as a spotlight focuses light • Display of synthesized forces and other haptic sensations to the kinesthetic senses • Devices, such as tracked gloves with pushbuttons, by which the user specifies interactions with virtual objects • Interaction techniques that substitute for the real interactions possible with the physical world • Examples of VR • A Desktop VR http://www.youtube.com/watch?v=Jd3-eiid-Uw • A CAVE VR http://www.youtube.com/watch?v=7_nUa4sFHSo&feature=related

5. Head-Mounted Display (HMD) • Head-Mounted Display (HMD) • A video display device mounted in a helmet, suspended one in front of each eye (in opaque HMDs) or projecting onto half-silvered mirrors in front of each eye (in see-through HMDs) Full immersion HMD http://www.darpa.mil/MTO/Displays/HMD/ Factsheets/immersion.html)

6. CAVETM • CAVETM • Provides the illusion of immersion by projecting stereo images on the walls and floor of a room-sized cube • Advantages • A wide surrounding field of view • The ability to provide a shared experience to a small group • Disadvantages • The cost of multiple image generation systems (although not a serious limitation nowadays) • Space requirement for rear projection • 4-8 feet or more, depending on the size of the screen • Brightness limitation due to large screen size • Result in scenes of approximately full-moon brightness and hinder color perception • Corner and edge effects that intrude on displayed scenes • An alternative is to use Dome systems in which imagery is projected onto a hemisphere surrounding

7. An illustration of DOME CAVE at NCSA (National Center for Supercomputing Applications) at UIUC (http://cave.ncsa.uiuc.edu/about.html)

8. Panoramic Displays • Panoramic Displays • One or more screens arranged in a panoramic configuration, or a single, curved screen on which images from multiple projectors are tiled together • Especially suit groups; multidisciplinary design reviews commonly use this type of display • One person drives the viewpoint • Issues • Edge blending, viewpoint-dependent distortion correction, viewpoint-dependent gain correction CURV™, by Fakespace Lab (http://www.fakespace.com/curv.htm)

9. Workbenches • Workbenches • Flat, rear-projection screens that display images in stereo and can be set up in a horizontal or tilted position M1 DeskTM, by Fakespace Lab (http://www.fakespace.com/M1Desk.htm) Responsive workbench, by Dr. Krueger at Stanford (http://graphics.stanford.edu/projects/RWB/)

10. Boom (Binocular Omni-Orientation Monitor) • Boom • A head-coupled stereoscopic display device that the screen and optical system are housed in a box that is attached to a multi-link arm; the user looks into the box through two holes, sees the virtual world, and can guide the box to any position within the operational volume of the device Boom3C, by Fakespace lab (http://www.fakespacelabs.com/tools.html)

11. Fishtank VR • Fishtank VR • A desktop VR system in which images are displayed on a desktop monitor, usually in stereo, and coupled to the location of the head which is tracked, resulting in the illusion of looking into a “fishtank” • Commonly applied in CAD and design areas where immersion is not of much significance Fishtank VR (http://www.faw.uni-linz.ac.at/save/)

12. Properties of VR Displays • Spatial Resolution • The ability of the system to spatially discriminate an object in the field of view; A system with higher resolution can resolve an image with smaller size • Because the smallest unit of an image is pixel, the resolution of a display is limited by its pixel size • Temporal Resolution • The time interval between images, or the number of frames captured per second • Often there is a tradeoff between spatial resolution and temporal resolution • Contrast • The ratio of the brightest part of an image to the darkest part of the image • Brightness • The perceived amount of light

13. Properties of VR Displays (Cont.) • Number of Display Channels • e.g. RGB channels, luminance channel • Focal Distance • Distance from the center of the lens to the point that is in focus • Opacity • The amount of transparency of the display • Field of View • The angular extent of the observable world that is seen at any given moment • Field of Regard • The amount of space surrounding the user that is filled with the virtual world

14. Motion Tracking • Usage • In VR, tracking technology is required to monitor the real-time position and orientation of the user’s head and limb • Mechanical Tracker • A simple mechanical tracker can take the form of mechanical arm attached to the tracked object • Very useful when integrated with a hand-held device • e.g. Boom3C • High accuracy and low latency due to its electromechanical nature • Restricted active volume (movement)

15. Motion Tracking (Cont.) • Optical Tracker • Infrared video cameras that record the movement of a person • Attached to the person is a collection of markers in the form of small balls fixed to a critical joints • When the moving person is illuminated with infrared light the marker balls are readily detected within the video images • Fast and low latency • The system depends on the line-of-sight, so the orientation of the cameras must ensure that the markers are always visible • Often prone to interference caused by ambient lighting conditions ARTTrack1 and ARTTrack2, by Advanced Realtime Tracking Inc. (http://www.ar-tracking.de)

16. Motion Tracking (Cont.) • Ultrasonic Tracker • Ultrasonic sound waves are used to locate the user’s position and orientation • Usually used for fishtank VR in which the ultrasonic tracker is placed on the top of the monitor and records the user’s head movements • Simple and low cost • Slow, restricted active volume, sensitive to temperature and depends on the line-of- sight Logitech Ultrasonic Head Tracker (http://www.i-glassesstore.com/logtractracs.html)

17. Motion Tracking (Cont.) • Electromagnetic Tracker • Employ a device called a source that emits an electromagnetic field, and a sensor that detects the radiated field • The source, which can be no bigger than a 2-inch cube, can be placed on a table or fixed to a ceiling • The sensor is even smaller and is readily attached to an HMD or fitted within a 3D mouse • Fast and very low latency; no light-of-sight restriction • Restricted active volume and are prone to interference of metallic objects miniBIRD, by Ascension Technology Corp. (http://www.ascension-tech.com/ products/minibird.php)

18. Interaction Devices • Usage • Allowing users to interact with virtual objects • SpaceMouse/SpaceBall • Hand-held device containing a tracker sensor and some buttons, used for navigating or picking objects within a VE • 6 DOF operations • Transitions in X, Y, Z axes and rotations around X (pitch), Y (yaw), and Z (roll) axes • Some allow zooming in/out objects SpaceMouseTM, SpaceBallTM 5000, by 3DConnexion Corp. (http://www.vrlogic.com/html/3dconnexion/3d_connexion.html)

19. Interaction Devices (Cont.) • Gloves • Gloves equipped with sensors that track the user’s hand movement • Enable natural interaction with objects • Modern VR gloves are used to communicate hand gestures (such as pointing and grasping) and in some cases return tactile signals to the user’s hand Pinch Gloves, by Fakespace lab (http://www.fakespacelabs.com/tools.html)

20. Haptic Devices in VR • Usage • A haptic device gives people a sense of touch with computer generated environments, so that when virtual objects are touched, they seem real and tangible • e.g. A medical training simulator in which a doctor can feel a scalpel cut through virtual skin, feel a needle push through virtual tissue, or feel a drill drilling through virtual bone • Current Technologies • Force feedback joystick • Virtual styluses • Sensable phantom series, by SensAble Technologies • Virtual gloves • Immersion cyber series, by Immersion Corp.

21. Force Feedback Joystick • Force Feedback Joystick • A device allowing the users to feel force of magnitude and orientation, aside from measurement of depression and twist of its stick Rumble Pak, by Nintendo (In most console video game systems today)

22. Sensable Phantom Series • By SensAble Technologies (http://www.sensable.com) • Positional sensing: X, Y, Z, pitch, roll, yaw • Force feedback: X, Y, Z • Range of motion: hand movement pivoting at wrist • Maximum force: 1.8 lbs • Intended for use in haptic research and free-form modeling Phantom desktop • Positional sensing: X, Y, Z, (pitch, roll, yaw with an additional separate encoder stylus gimbal) • Force feedback: X, Y, Z • Range of motion: hand movement pivoting at wrist • Maximum force: 1.9 lbs Phantom premium 1.0

23. Positional sensing: X, Y, Z, (pitch, roll, yaw with an additional separate encoder stylus gimbal) • Force feedback: X, Y, Z • Range of motion: lower arm movement pivoting at elbow • Maximum force: 1.9 lbs Phantom premium 1.5 • Positional sensing: X, Y, Z, (pitch, roll, yaw with an additional separate encoder stylus gimbal) • Force feedback: X, Y, Z • Range of motion: full arm movement pivoting at shoulder • Maximum force: 4.9 lbs Phantom premium 3.0

24. Pros and Cons of Virtual Styluses • Pros • Inexpensive • Easy to set up and operate • Works on a desktop • Well suited for remote manipulation • Cons • Not immersive • Haptic response at a single point only

25. Virtual Gloves • Immersion cyber series, by Immersion Corp. (http://www.immersion.com/) • Senses position of finger; no force feedback CyberGloveTM • Adds tactile feedback to CyberGlove using vibrations on fingertips or palm • Limited to simple pulses or sustained vibration CyberTouchTM

26. Force feedback for fingers and hand CyberGraspTM • Force feedback for hand and arm • Can be used together with CyberGrasp CyberForceTM

27. Pros and Cons of Virtual Gloves • Pros • Multiple points of haptic and tactic responses • Allows for full immersion with HMDs • Cons • Expensive • Difficult to set up and operate

28. Production-Stage Applications of VR(Brooks, 1999) • Vehicle Simulation • Ergonomics Evaluation and Design • Training and Experience

29. Vehicle Simulation • Vehicle Simulation • Was the first application of VR and is still the most advanced • Cases • 747 simulator at British Airways • Merchant ship simulation at Warsash Maritime Center Ship bridge simulator at Warsash Maritime Center

30. Vehicle Simulation (Cont.) • Lessons Learned • Why are VR vehicle simulators useful? • They are much cheaper to build than the real vehicles • They make it possible to thoroughly train operators in extreme situations and emergency procedures where real practice would imperil equipment and lives • Scenarios can be easily run and modified, enabling more efficient training • What aspects of VR make it work so well? • Immersion is complete (or nearly complete) • The near-field haptics are perfect • What aspects of VR are critical to success? • Realism of the graphics • Realism of the sound • Realism of the haptics • Realism of the motion • Realism of the interaction (how does the display respond to the user’s actions?)

31. Ergonomics Evaluation and Design • Cases • Evaluating ergonomics in cars, at Daimler-Chrysler Technology Center • Submarine design at General Dynamics • Design review at John Deere View of virtual wind-shield wiper visibility at Daimler-Chrysler’s Technology Center

32. Ergonomics Evaluation and Design (Cont.) • Lessons Learned • Why is VR useful for ergonomics evaluation and design? • Facilitate communication of ideas among team members • Save the cost of materials used to develop physical prototypes • Speed up design review and change cycles • What aspects of VR are critical to success? • True scales of the modeled objects • The farther the design gets from its conceiver, the better the visuals need to be, in order to enable a factory-floor foreman or an operations person to get an accurate, internalized perception of the design

33. Training and Experience • Cases • Astronaut training at NASA-Huston • Psychiatric treatment at Georgia Tech and Emory University Medical School • Fear of flying • Fear of heights • Fear of public speaking • Post-traumatic stress disorder for Vietnam War veterans

34. NASA-Houston’s “Charlotte” (a haptic simulator) virtual weightless mass lets astronauts practice handling weightless massy objects • (a) The psychologist gently leads to the patient into a simulated Vietnam battle scene • (b) Imagery seen by the patient (a) (b) Vietnam War simulation at the Atlanta Veterans Administration Hospital

35. Training and Experience (Cont.) • Lessons Learned • Why is VR useful for training and experience? • For NASA, it offers the ability to simulate unearthly experiences • e.g. flying about in space using the back-mounted flight unit which is designed principally as an emergency device for use if an astronaut’s tether breaks; moving around on the outside of a space vehicle • For psychiatry, it can save cost (both money and time) and offers a “safe” form of exposure to traumatic stimuli • What aspects of VR are critical to success? • Immersion • Haptics • Sound

36. Open Challenges (Brooks, 1999) • Technological • Lowering latency to acceptable levels • Rendering massive models in real time • Choosing which display best fits each application (HMD, CaveTM, benchmark, or panorama) • Producing satisfactory haptic augmentation for VR illusions • Systems • Interacting more effectively with virtual worlds • Manipulation • Specifying travel • Wayfinding • Making models of worlds efficiently • Modeling the existing worlds • Modeling the non-existing worlds • Measuring the illusion of presence and its operational effectiveness

37. References • Brooks Jr., F.P. (1999). What’s Real About Virtual Reality. IEEE Computer Graphics and Applications,19(6), 16-27. • Sherman, W.R., & Craig, A.B. (2003). Understanding Virtual Reality. San Francisco, CA: Morgan Kaufmann.