MULTIMODAL EMOTION PERCEPTION: ANALOGOUS TO SPEECH PROCESSES

2. MULTIMODAL EMOTION PERCEPTION: ANALOGOUS TO SPEECH PROCESSES Dominic W. Massaro

3. Old Fashioned View of Spoken Language and Communication

4. Anecdotal Evidence for Functional Value of Visible Speech • Persons with Hearing Loss • Benjamin Franklin in France • Hal in 2001: A Space Odyssey • “Hear TV Better with Glasses On” • Poorly Dubbed Foreign Films

6. Visual Communication in Love

7. Value of Talking Heads • Enhance Realism and Naturalness • Convey Intention and Emotion • Enhance Intelligibility

9. Research Strategy to Develop and Evaluate the Effectiveness of Talking Heads • Auditory Synthetic speech • Computer Animated Talking Head • Development and Evaluation

10. PSLTalk (Baldi) • Successive Approximations to Realism • Real Time on PC Platforms • Controlled by text-to-speech synthesizer

11. BaldiDescribesHimself

12. Rotation of points • movement around axis, e.g., jaw rotation

13. Translation • movement of points, e.g., raise upper lip

14. Interpolation • Between two different subsections of wireframes--e.g., neck size

15. Scaling • constant multiplier, e.g., head width

16. Features • Driven by Text to Speech Engine • Target Values for each phoneme • Coarticulation

20. Additional Features • Coarticulation • Paralinguistic Properties • Nonspeech Segments • Texture Mapping

21. Paralinguistic Synthesis • Nonspeech Segments • Breadth Noise, Cough, Clear Throat, Laugh, Lip Smack, Sneeze, Tongue Click, Burp Baldi’snonspeech

22. Texture Mapping

24. PSLTalk (Baldi) • Alignedwithnaturalauditory speech

25. Evaluating Intelligibility of Baldi, our Talking Head • Speechreading Syllables and Words • Understanding Sentences in Noise • Compare Baldi to Humans

27. Additional Features • Tongue and Palate • Can Hide Noncritical Components • Can Reveal Normally-Hidden Parts • Can Highlight Interaction of Articulators

28. Training with Baldi’s New Tongue and Palate

29. Language Training exercise lay, ray, they Top View Side View

30. Language Training exercise wah, rah Back View

31. Synthesis of Emotion • Voice is Informative • Face is More Critical • Basic Universal Emotions • Happiness, Anger, Surprise, Fear, Disgust, and Sadness, and Neutral • Can specify the degree to which Baldi expresses each of these emotions, and some combination

32. Can specify the degree to which Baldi expresses each of these emotions, and some combination Over 80% correct in 6 alternative task

33. Emotion Training exercise surprise Front View

34. Emotion Hallucinations

35. Speech Hallucinations

36. Experimental Strategy to Study How Emotion is Processed • Manipulate auditory and visual speech • present unimodal stimuli • present factorial bimodal stimuli • no feedback in task • test models of perception

37. Expanded Factorial Design (Baldi) • Play Continuum-- voice fastest moving • Play Continuum-- face fastest moving

44. Two Sources Better than either One Alone

45. Unique Response from Two Sources

46. Brow 2 3 4 Happy none Angry 2 3 4 Happy none Angry Mouth

47. Expanded Factorial Design (Baldi) • Play Continuum-- brow fastest moving • Play Continuum-- mouth fastest moving

48. Identification Judgments

49. Rating Judgments