CSCI480/582 Lecture 31 Chap 5.3 Performance Driven Facial Animation Apr, 13, 2009

1. CSCI480/582 Lecture 31 Chap 5.3 Performance Driven Facial Animation Apr, 13, 2009

2. Outline • What differs facial animation from body animation? • Face model • Facial motions and FACS • Facial motion synchronization with audio production • Performance-driven techniques

3. Face Model for Facial Animation • In modeling face, Four aspects needs to be considered • Structure: vertex distribution and topology to represent face in geometry • Appearance: color, texture, light reflection of face

4. Face Model - Geometry • Dense mesh around the “holes” • The density of the mesh determines the level of details in facial deformation • Eyes, ears, and mouth cavity • Teeth and tongue

5. Face Model - Appearance • Using Texture Mapping • Unwinding the “cylinder” surface by concatenating multiple camera views to form the texture map

6. More Advanced Skin Appearance Model • The skin is translucent • Light is scattered through multiple layers, then reflected from a sub-layer • Bidirectional Reflectance Distribution Function is often used

7. Bidirectional Reflectance Distribution Function(BRDF) • A 4-dimensional function that defines how light is reflected at an opaque surface • Input: • incoming light direction wi • Outgoing direction wo • Output: • Ratio of reflected radiance along wo to irradiance incident along wi

8. Facial Motion • Expressions involving • All front facial mesh • Eye ball orientation • Speeches involving • Lip and jaw • Tongue • The combination of expressions and speeches

9. Facial Acting Code System • Originally developed by Ekman and Friessen in 70s • Based on the physiology of facial muscles • A common standard to systematically categorize the physical expression of emotions

10. FACS – The Codes for Action Units • Action Descriptors • Inner Brow Raiser -- Frontalis (pars medialis) • 2 Outer Brow Raiser -- Frontalis (pars lateralis) • 4 Brow Lowerer -- Corrugator supercilii, Depressor supercilii • 5 Upper Lid Raiser -- Levator palpebrae superioris • 6 Cheek Raiser -- Orbicularis oculi (pars orbitalis) • 7 Lid Tightener -- Orbicularis oculi (pars palpebralis) • 9 Nose Wrinkler -- Levator labii superioris alaeque nasi • 10 Upper Lip Raiser -- Levator labii superioris • … • 66 Action descriptors • Demo

11. Lip-Sync by Viseme Morphing • Use reference video to rotoscope the shape of the 3D model for key gestures • Use a set of viseme corresponding to phonemes, then interpolate between viseme gestures • Such approach often is not detailed enough for finer visual speech animation

12. Facial Motion Driven by Mocap Data • Captures subtle details of facial expressions • But, Mocap Data need be to cleaned for good rendering effect. Often this step is labor intensive. • How can we make Mocap more flexible?

13. FACS-Viseme Solver • Use a small set of Mocap data to synthesize a much larger range of facial motions • Each piece of mocap data in the small recording set represent an Action Unit • Use Action Unit analysis to decompose recorded Mocap data for easy post editing • Use audio-visual mapping analysis to align mocap segment for new speech