Cross-modal perception of motion-based visual-haptic stimuli

1. Cross-modal perception of motion-based visual-haptic stimuli Ian Oakley & Sile O’Modhrain Palpable Machines Research Group http://www.mle.ie/palpable ian@mle.ie sile@media.mit.edu Media Lab Europe Sugar House Lane Bellevue Dublin Ireland

2. Overview • Background – Touch in Broadcast Media • The Touch TV Project • Role of cross modal perception in TTV • Experimental Study - Pilot • Conclusions and Future Work

3. Touch in Broadcast Media • Adding haptic information to a broadcast may enhance viewing • Motivation • Consumer demand for richer media • Led to advances in audio and video technologies • Touch may provide greater richness • Latest advances not enhanced viewing experience • e.g. digital and interactive TV • Evidence supporting this assertion • Touch in virtual environments linked to increased ratings of presence and co-presence • Adoption of touch in gaming and simulator systems

4. The Touch TV Project • Ongoing work investigating integration of touch into broadcast content • Currently, focused on a practical and theoretical exploration of the design space available for this work • Involves a number of research threads ranging from authoring of media to psychological study

5. Cross-Modal Perception in TTV • Touch TV inherently cross-modal • Involves construction of immersive and involving media with graphics, audio and touch components • Techniques for combining graphics and audio established in this context • E.g. visual capture • How to add touch to this format is an open question • How do we add haptic information to an audio-visual stream ensuring that it will be interpreted as we intend?

6. Experimental Study • Interested in combination of visual and haptic stimuli representing movement in complex graphical scenes • Relatively complex stimuli • Initial questions: • Can haptic cues be associated with visual movement? • What is the effect of visual distractors? • What is the effect of visual target position? • What kind of haptic cues should we use? • Attempted to address these questions in a pilot study

7. Experimental Study • In each trial, the visual display consisted of a number of bouncing balls controlled by a simple mathematical model • Each ball had a different spring constant – different “bounciness” and was subject to gravity • Force displayed on a PHANToM haptic interface either represented the velocity of one of the displayed balls, or of an unseen ball • Participant’s task was to determine whether or not what they could feel related to one of the displayed balls

8. Experimental Study • Varied: • Number of distractors (5 possibilities) • Target spring constant (6 possibilities) • Target position (5 possibilities) • Haptic-Visual match (2 possibilities) • Total of 300 trials presented in random order • Practice consisted of a random selection of 50 trials • 10 Participants • All employees at Media Lab Europe • 6 male, 4 female, all right-handed, none reported tactile or visual impairment, range of familiarity with haptic devices (2.5 on a scale from 0-5) • Participants wore noise cancelling headphones • Were able to rest between each trial

9. Experimental Study • Gathered trial time, error rate • Trial time for visual presence/absence of haptic stimuli: • T-test revealed no significant differences

10. Experimental Study • Further analysis restricted to trials in which haptic ball was visually present • Used ANOVA, post-hoc t-tests with Bonferroni CI adjustments • Did not include trials in which subject made errors in temporal analysis • However, relatively large error rate • Collapsed position factor for temporal analysis • Used a 5 (distractors) x 6 (spring values) • No interactions found (F=0.399, p=0.991) • Error data 5 (distractors) x 6 (spring values) x 5 (positions) • One interaction: Distractors by Spring Value (F=1.974, p<0.01)

11. Results • Trial time for differing numbers of visual distractors • ANOVA: F=8.235, p<0.001 Results of post-hoc t-tests

12. Results • Trial time for differing spring values • ANOVA: F=83.881, p<0.001 Results of post-hoc t-tests

13. Results • Error rate for differing numbers of distractors • ANOVA: F=20.614, p<0.001 Results of post-hoc t-tests

14. Results • Error rate for differing spring values • ANOVA: F=4.861, p<0.001 • Post-hoc t-tests revealed 0.15 spring value led to significantly higher error rate than 0.4 spring value

15. Results • Error rate for target positions • ANOVA: F=0.399, p<0.05 • Post-hoc t-tests did not attain significance

16. Interpretation • Subjects merged visual-haptic motion stimuli • Error rate always less than chance • Increase in trial time and error rate with increased numbers of distractors indicates serial search • Haptic target did not “pop out” as in parallel search • Equality of present and absent results suggests subjects performed an exhaustive search • Possible speed/accuracy trade off • Errors decreased, time increased with larger spring values • Eccentricity effect in position error data suggests subjects focused on the centre of the screen

17. Conclusions and Future Work • Study successfully informed basic questions relevant for our TTV domain • Relevant to our authoring process • Many questions remain • Contrast same scenario with visual/audio and visual/audio/haptic stimuli • Role of visually distinct objects • Is mapping force to motion the best solution? • Basic scaling research on force presentation in this domain • …

18. Cross-modal perception of motion based visual-haptic stimuli Ian Oakley & Sile O’Modhrain Palpable Machines Research Group http://www.mle.ie/palpable sile@media.mit.edu ian@mle.ie Media Lab Europe Sugar House Lane Bellevue Dublin Ireland EuroHaptics 2003 6th-9th July www.mle.ie/palpable/eurohaptics2003