Combining Sampling and Autoregression for Motion Synthesis

1. Combining Sampling and Autoregression for Motion Synthesis David Oziem, Neill Campbell, Colin Dalton, David Gibson and Barry Thomas

2. Project Group • Motion Ripper Project • Methods of motion capture. • Re-using captured motion signatures. • Synthesising new or extend motion sequences. • Tools to aid animation. • Collaboration between University of Bristol CS, Matrix Media & Granada. http://www.cs.bris.ac.uk/Research/Vision/MotionRipper/

3. Aim • A common presumption in synthesis is that the original data is stationary, i.e. • Its statistical properties hold true throughout the signal. • Given a random point in time it should not be possible to exactly predict its origin.

4. Example Original candle flame • Candle frame includes multiple different ‘states’ of motion. • We would like to be able to ignore these states. • Reisell’00 used prior knowledge (i.e. labelled frames)

5. Aim • Synthesise non-linear, non-Gaussian, non-stationary motion signals in N-D. Primary mode from Principal components analysis (PCA)

6. Video Textures • Video textures or temporal textures are textures with motion. (Szummer’96) • Schodl’00, reordered frames from the original to produce loops or continuous sequences. • Doesn’t produce new footage. • Campbell’01, Fitzgibbon’01, Reissell’01, used Autoregressive process (ARP) to synthesis frames. Examples of Video Textures

7. Autoregressive Process • Statistical model • Calculating the model involves working out the parameter vector (a1…an) and w. Current value at time t y(t) = – a1y(t – 1) – a2y(t – 2) – … – any(t – n) + w.ε Noise Parameter vector (a1,…,an)

8. Copying and ARP • Combine the benefits of copying with ARP • New motion signatures. • Requires no prior knowledge. • Handles non-Gaussian distributions and non-stationary sequences.

9. Important to reduce the complexity of the search process. Copying and ARP Original input PCA Reduced input

10. Temporal segments of between 15 to 30 frames. Need to reduce each segment to be able to train ARP’s. Copying and ARP Original input PCA Segmented input Reduced segments PCA Reduced input

11. Many of the learned models are unstable. 10-20% are used. Copying and ARP Original input PCA Segmented input Reduced segments PCA ARP Reduced input Synthesised segments

12. Copying and ARP Original input PCA Segmented input Reduced segments PCA ARP Reduced input Segment selection Synthesised segments Outputted Sequence

13. Example First mode Possible segments. End of generated sequence. Compared section Time t

14. Example First mode The segment to be copied is randomly selected from the closest 3. Time t

15. Example First mode Segments are blended together using a small overlap and averaging the overlapping pixels. Time t

16. Results Candle flame

17. Results Barman

18. Results Copying & ARP model original generated

19. Results Copying & ARP model original generated

20. Results Copying & ARP model original generated

21. Results original

22. Results Copying & ARP model original generated

23. Conclusions • Motion synthesis of complex sequence is possible with this approach. • New unseen footage/motion can be synthesised. • The model occasionally gets stuck in loops. • Use a hierarchical model to guide the sampling process.

24. Thanks • Questions.