Brett Allen 1,2 , Brian Curless 1 , Zoran Popovi ć 1 , and Aaron Hertzmann 3

1. Learning a correlated model of identity and pose-dependent body shape variation for real-time synthesis Brett Allen1,2, Brian Curless1, Zoran Popović1, and Aaron Hertzmann3 1 University of Washington 2 Industrial Light & Magic 3 University of Toronto

2. movies games telepresence design Motivation

3. + Identity pose shape Goal • We would like to be able to generate body models of any individual in any pose. - want to synthesize models in real-time - model should be learnable from real data

4. Data • CAESAR data set: 44 subjects in 2 poses • Multi-pose data set: 5 subjects in 16 poses • Dense-pose data set: 1 subject in 69 poses [Anguelov et al. 2005]

5. Related work

6. Anatomical methods Chadwick et al. 1989 Turner and Thalmann 1993 Scheepers et al. 1997 Wilhelms and Van Gelder 1997 … Aubel 2002

7. v = shape vector p = example parameters  = function parameters Example-based methods Given: n examples v0…vn and n sets of parameters p0…pn (optional) find .

8. 1 2 2 1 3 4 4 3 Allen et al. 2002 Scattered data interpolation columns of  = key shapes i = reconstruction weights from applying k-nearest neighbors or RBFs on p Lewis et al. 2000 Sloan et al. 2001 Kry et al. 2002

9. An aside on enveloping Enveloping + scattered data interpolation = “corrective enveloping”

10. Blanz & Vetter 1999 Latent variable models x = latent variable (component weights) W = components in columns v = average shape Allen et al 2003 Seo et al 2003 Anguelov et al 2005 …

11. Pose variation vs body variation Sloan et al. 2001

12. Pose variation vs body variation Anguelov et al. 2005

13. Intrinsic skeleton parameters: bone lengths and carrying angles Our approach c = “character vector”: all information needed to put a character in any pose v = shape in a particular pose

14. Two Problems 1. Scans might not be at “key” poses 2. Scans are not complete Maximize: p(c | {e()}) …actually, we don’t know the pose or skinning weights either: Maximize: p(c, s, q{} | {e()})

15. Maximum a posteriori estimation

16. Results

17. Going to multiple characters • One possibility: Learn several character vectors separately, then run PCA. • Two problems: • the character vector contains values that have different scales (rest positions, offsets, bone lengths) • we don’t have enough data!

18. Identity variation ~ ~ c is the character vector of the th example person g(c,s,q) applies skinning and pose space deformations

19. Alternating optimization • We initialize the {x} with the weights from running PCA on estimated skeleton parameters. • We then optimize W, c, s, q. • Then we optimize for {x}. • Repeat…

20. Results

21. Results (video)

22. Conclusions We present a flexible approach for learning body shape variation between individuals and between poses, including the interrelationship between the two. + very general: can handle irregular and incomplete sampling in regard to both the poses/identities scanned, and in the surfaces themselves + the learned model can generate body shapes very quickly (over 75 fps)

23. Limitations • You need a lot of data! Our data set was too sparse in some areas. • Some poses are hard to capture. • It’s very hard to compensate for the skinning artifacts. • The shape matching could be improved (high-frequency details are lost if the matching is poor).

24. Acknowledgements • UW Animation Research Labs • Washington Research Foundation • National Science Foundation, NSERC, CFI • Microsoft Research, Electronic Arts, Sony, Pixar • Kathleen Robinette and the AFRL lab • Dragomir Anguelov • Domi Pitturo