1 / 27

Towards a virtual agent using similarity-based laughter production

Towards a virtual agent using similarity-based laughter production. Jérôme Urbain, Stéphane Dupont, Thierry Dutoit, Radoslaw Niewiadomski, Catherine Pelachaud TCTS Lab, Faculté Polytechnique de Mons CNRS - LTCI UMR 5141, Institut TELECOM - TELECOM ParisTech. Context.

paulos
Télécharger la présentation

Towards a virtual agent using similarity-based laughter production

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Towards a virtual agent using similarity-based laughterproduction Jérôme Urbain, Stéphane Dupont, Thierry Dutoit, Radoslaw Niewiadomski, Catherine Pelachaud TCTS Lab, Faculté Polytechnique de Mons CNRS - LTCI UMR 5141, Institut TELECOM - TELECOM ParisTech

  2. Context • Desire to communicate with machines as we do with humans • Lack of naturalness, expressivity, perception of emotions • Laughter is a very important signal: • Conveys emotional contents • Communicative • ... <footer>

  3. Objectives • Improve the expressivity of an Embodied Conversational Agent (ECA) by enabling it to laugh: • acoustic laughter production • synchronisation with virtual agent expressions • ability to instantaneously answer to a laughter <footer>

  4. Outline • Multimodal expressions of emotions by an ECA • Acoustic laughter production based on similarities • Upcoming projects <footer>

  5. Multimodal expressions of emotions • Go beyond « static images »/linear expressions (e.g. beyond basic emotionspredictions) • Complete data obtainedfromtheory and literature • Multimodal expressions of emotions • across modalities • many emotions are expressed by sequences (or combination) of multimodal signals rather than monomodal signals (eg static facial expressions) D. Keltner, B. N. Buswell Embarrassment: Its Distinct Form and AppeasementFunctions

  6. Multimodal expressions of emotions • Difficulty: lack of relevant research and video-corpora • new video-corpus of examples of spontaneous multimodal behaviors (based on TV broadcasts) • annotation in Anvil (Kipp 2001) • Specification of Behaviors Sets and Constraints

  7. Corpus of emotional displays • Annotation of audio-visual recordings from reality shows, hidden camera recordings, Belfast Naturalistic database, EmoTV corpus. • The observed people are non actors in emotional situations: natural and not stereotyped multimodal behaviour is displayed • 20 video clips (3 to 14 seconds each): • relief (2), tension (6), joy (2), sadness (2), anger (3), despair (1), fear (4)

  8. Multimodal annotation scheme • Annotation on Anvil v4.7.6 (Kipp, 2001) with 5 tracks: • Emotion (inferred from the situation) • Facial expression (FACS coding) • Head movement • Gaze movement • Gesture

  9. Multimodal Joy annotation Video annotations with the Anvil software: multimodal display of joy from the Belfast Naturalistic Emotional Database (Cowie et al., 2003)

  10. Multimodal Joy annotation • Joy: • arm movements, • head/torso movements towards front and backwards, • tilts, and micro tilts, • movements to the side, • Great arm movements (like playing on a drum) • Facial expressions like smile and raise eyebrow. Humaine video-corpus

  11. Multimodal expressions of emotions • formalization of multimodal expressions of emotions by constraints FML: Emotion label Constraints: - time constraints - ordering - simultaneity - proba of occurrence Emotion Behavior Sets: (set of signals of different modalities) Annotation of corpus variety ofmultimodal expressions of emotions

  12. Multimodal expressions of emotions • literature constraints <multimodal emotion="embarrassment"> <signals> <signal id="1" name="head=head_down_strong" repetitivity="0" min_duration="2" ... <signal id="2" name="head=head_left_strong" repetitivity="0" min_duration="5" ... <signal id="3" name="gaze=look_down" repetitivity="0" min_duration="2" ... <signal id="4" name="gaze=look_right_strong" repetitivity="0" min_duration="1" ... <signal id="5" name="gaze=look_left_strong" repetitivity="0" min_duration="1" ... <signal id="6" name="affect=smile" repetitivity="1" min_duration="2" ... ... </signals> <cons> <con type="minus"> <arg id="6" type="start"/> <arg id="2" type="start"/> <lessthan value="0"/> </con> <con type="minus"> <arg id="7" type="start"/> <arg id="2" type="start"/> <lessthan value="0"/> </con> .... D. Keltner, B. N. Buswell Embarrassment: Its Distinct Form and Appeasement Functions annotation

  13. Multimodal expressions of emotions • implementation of the algorithm <emotion id="e1" type=“joy" start="1.0" end="14" /> BML A set of behaviours FML animation FMLRealizer BMLRealizer

  14. Example Embarrassment Embarrassment Joy

  15. Video

  16. Outline • Multimodal expressions of emotions by an ECA • Acoustic laughter production based on similarities • Upcoming projects <footer>

  17. Audio Cycle • A prototype application for browsing through musical loop libraries. AudioCycle provides the user with a graphical view where the audio extracts are visualized and organized according to their similarity in terms of musical properties, such as timbre, harmony, and rhythm. The user is able to navigate in this visual representation, and listen to individual audio extracts, searching for those of interest. • Richer in features than other similar concepts we have seen.

  18. Technologies & Architecture Audio, Musical Loops 3D Audio Rendering Audio Analysis Visualization 3D Visual Rendering Meta-Data & Features User Input

  19. Extractedfeatures • Timbre: Mel-Frequency Cepstral Coefficients • Harmony and Melody: Chromas (information about the notes played) • Rhythm: periodicity, « Beats per Minute »

  20. Visualization: Clustering Loop Cluster Centroïd Reference Loop

  21. Adaptation to laughter • Firsts tests: Audio Cycle as it is: • some grouping of classes (whisper-like, « retained » laughters, melodious laughters, …) • some grouping of classes • laughing audience

  22. Adaptation to laughter • New feature set (« laughter »): • mean pitch • rate of voiced frames • mean energy • maximum aplitude • duration • Automatic laughter bursts segmentation

  23. Adaptation to laughter • First laughter synthesis using concatenation of bursts

  24. Outline • Multimodal expressions of emotions by an ECA • Acoustic laughter production based on similarities • Upcoming projects <footer>

  25. Future Work • Adapt Audio Cycle to Laughter Laughter Cycle: • improve features • model for concatenation of bursts • how to combine laughters

  26. Future Work • Audovisual laughing machine: eNTERFACE’09 • synchronisation between audio and ECA expressions, using emotional behavior descriptors • automatic answer to an input laughter

  27. Future Work • Numediart Research Program: http://www.numediart.org • eNTERFACE’09: http://www.infomus.org/enterface09/ • CALLAS project: http://www.callas-newmedia.eu/

More Related