1. Multimodal Virtual Environments:Response Times, Attention, and Presence B93902033陳柏叡

2. Introduction • Efficiently combine sensory information from two or three channels • Three channels: vision, audio, haptic • Differentiates from older technologies - communicating only via a single channel • The underlying cognitive mechanisms are still elusive and unknown

3. Environments Richness Results in a Complete and Coherent Experience • Single channel conveys limited and insufficient information to the senses - Engenders a lower sense of presence • Multimodal VE provide a greater extent of sensory information to the observer - Sense of being present is felt stronger

4. Multimodal VE are Mimicking Reality Better • Many of daily experiences in the real world are fundamentally multimodal • Grasping an object – visual, haptic • Gastronomic pleasure – taste, smell, vision • Multimodal VE have a clear advantage, in mimicking a multimodal phenomenon

5. Better Integration and Filling in of Missing Information • Synthetic VEs provide fewer sensory cues than physical environments • User needs to interpolate sensory stimuli to create a functional mental modal • Result in an enhanced sense of presence

6. Faster Mental Processing Enables Deeper and Richer Experience • Using Reaction Time to compare the brain processing speed of unimodal, bimodal, and trimodal signals • Faster processing at the perceptual stage allow users more time in consequent cognitive stages • Contributing to the great sense of presence

7. Experimental Design • Materials – touch-enabled computer interface, haptic device(force-feedback) • Participants – 16 students(unaware of the purpose of the experiments) • Under the guidelines of the ethical committee and with its approval

8. Stimuli • Holding the stylus in their hand as if they were writing • On each trail the computer generated a sensory stimulation – uni, bi, trimodal • Visual – changing color • Auditory – compound sound • haptic – resisting force

9. Procedure • Pressing a button on the stylus as soon as they detected the stimuli • Computer generated the stimulation randomly • Consisted of six blocks of trails (three with dominant hand, three with the other hand) • Each of blocks consisted of 105 single trail, in which each of the seven conditions

10. Result-Unimodal • V – dominant hand 430ms (SD=94) not dominant 436ms (SD=92) • A – dominant hand 330ms (SD=103) not dominant 326ms (SD=76) • H – dominant hand 318ms (SD=99) not dominant 334ms (SD=91)

11. Result-Bimodal • VA – dominant hand 302ms (SD=78) not dominant 304ms (SD=70) • HV – dominant hand 294ms (SD=75) not dominant 306ms (SD=77) • HA – dominant hand 272ms (SD=81) not dominant 280ms (SD=69)

12. Result-Trimodal • dominant hand 430ms (SD=94) not dominant 436ms (SD=92) • Between-Hand comparisons - Revealed insignificant differences in the VA and HA conditions, and only marginal difference in HV conition

13. Multimodal Stimulation and Attention • Multisensory enhancement is modulated by attention • Hypothesizing that the brain allocates a greater amount of attention to multimodal and employing a larger neural network

14. Attention and Presence • The link between attention and presence is already documented in literature

15. Conclusion • At the initial perceptual stage – • Multimodal information is processed faster • Activating larger neural network, increasing user’s attention • At the consequent cognitive stages, more time to • Acquire a wider range of details and subtle for the display • Fill in missing information • Integrate all these informative cues • Contributing to a greater sense of presence