Robust global motion estimation and novel updating strategy for sprite generation

1. Robust global motion estimation and novel updating strategy for sprite generation IET Image Processing, Mar. 2007. H.K. Cheung and W.C. Siu The Hong Kong Polytechnic Univ. (香港理工大學)

2. Outlines • Overview / Introduction • Proposed system • New global motion estimation • Combing short- and long-term estimation • Dynamic reference frame • 2-pass sprite blending • Preserving frame resolution loss • Sprite updating • Overcoming illumination variations & object changing • Experimental results • Conclusions

3. Overview

4. Overview • Sprite • High resolution image • Composed of information belonging to an object visible throughout a video sequence • Background of a scene

5. Overview • Sprite background of frame 20 Sprite(Dimension: 2670x1072) background of frame 1(Dimension: 352x288)

6. Overview • Core of sprite generation • Global motion estimation (GME) • Finding a set of parameters representing camera motion between frames • Image registration • Iterative minimization • Blending • Temporal (weighted) averaging, median, updating

7. Introduction

8. Introduction • Global motion estimation • Image registration • Short-term motion estimation • Estimation between consecutive frames • Easy and accurate • Long-term motion estimation • Estimation between frames with temporal distance • Harder • Required to perform sprite coding • Single sprite for all frames in sequence

9. Introduction • Global motion estimation (cont.) • Short- to long-term estimation • Converting short-term motion parameters to long-term parameters • Error propagation • Directly long-term estimation • Estimation every frames directly to a specified base frame (reference frame) • No error propagation • Search range may be huge • Hard to find overlapping area

10. Introduction • Global motion estimation (cont.) • Hierarchical estimation • Rough estimation to find coarse parameters • Refining parameters • Using coarse parameters as initials • Iterative minimization • Some existing methods • Dufaux and Konrad • Szeliski • Smolic et. al. • Lu et. al.

11. Introduction • Restrictions • Background must be really static • Background objects must be still • No illumination variations • Dynamic sprite

12. Introduction • Classification • Static sprite • Build offline before coding individual frames • Quality degradation as frame increases • Motion estimation errors • Illumination variations • Background object changes • Dynamic sprite • Built dynamically online in both encoder and decoder while coding individual frames • Sprite is updated using reconstructed frame • Short-term estimation is employed • Error accumulated

13. Introduction • Proposed system • New global motion estimation • Directly estimating the relative motion between current image and a chosen reference frame • Give accurate, stable and robust estimation • Alleviate error accumulation • Hierarchical 3-levels approach • Coarse-to-fine approach • Sprite updating • Updating sprite only if necessary • Sprite update frames are generated and sent

14. Proposed system

15. Proposed system • Short-term GME to long-term GME More Error Registration Error Am1 + A(m+1)1 A(m+1)m Registration Error A(m+1)k = A(m+1)m Am1 GME = A(m+1)m Am(m-1) …  A21 Frame 1 A11 Frame m Am1 Frame m+1 …… reference frame Registration errors are ACCUMULATED

16. Proposed system • Directly measure to reference frame GME A(m+1)1 Registration Error Registration Error initial guess Am1 Frame 1 A11 Frame m Am1 Frame m+1 …… reference frame Registration errors are COMPENSATED

17. Proposed system • Weakness • Reference frame is temporally far from current frame • Frame contents may change largely • Background objects activities • Lighting conditions changes • Overlapping area could be smaller • Unfavorable to GME

18. Proposed system • Combining the advantages • Dividing video into groups of consecutive frames • 1st frame of each group is selected as reference • Frames in a group • Each frame is directly measured to the 1st frame • Smaller registration error • Merging groups • GMEs of reference frames of all groups are merged • Registration error is slightly increased R1 …… R2 …… R3 A(R2)(R1) A(R3)(R2) A(R1)(R1) + + A(R2)(R1) A(R3)(R1)

19. Proposed system • Proposed GME structure MotionEstimation A(m+1)k Frame z Amk Frame k Ak1 Frame m Amk Am1 Frame m+1 …… Chosen to bereference frame

20. Proposed system • Dynamic reference frame • 1st frame is the initial reference frame • Assigning current frame as new reference frame if • Displaced frame difference between registered current frame and the reference frame it large • Reference frame is not like current frame • Relative displacement between current frame and the reference frame is large • Overlapping area is too small or where Nr is a parameter between 0 and 1 (Nr=0.1 in practical)

21. Proposed system • Advantages • Accuracy • Accurate than short-term and directly long-term estimation • Very few memory usage • Estimations are performed frame-to-frame • Sprite building is not necessary

22. Proposed system • GME Reference frame(frame k) Frame z Three step search Block-based partialdistortion search Fast gradient method + A(m+1)k Amk

23. Proposed system • Motion model • Perspective motion model • 8 motion parameters to be determined • Three-step matching • 3-level pyramids for frame k and z are built using Gaussian down-sampling filter [¼, ½, ¼] frame k: reference frameframe z: transformed current frame m+1

24. Proposed system • Block-matching • Affine parameters are estimated by solving over-fitting equations • Results of block-based motion estimation are used to construct the equations • Parameter estimation • Fast gradient descent method by Keller and Averbuch where

25. Proposed system • Two-passed blending to avoid resolution loss • First pass: 1st frame as base frame • All frames are projected into 1st frame • Frame with minimal area of projected frame is selected as new base frame • Avoiding resolution loss • No real pixel blending applied • Second pass: new base frame • All frames are projected into new base frame • Simple temporal average blending • With bilinear interpolation

26. Proposed system • Dynamic sprite updating • Overcoming illumination variations • Single value in sprite can not represent intensity variations over the time • Accumulation of GME error blurring the frame • GME error in a reference frame will inherit into all of frames in the group

27. Proposed system • Studying the generated intensity error a pixel fromhomogeneous area a pixelfrom texture area an edge pixel # of pixel withsignificant error translation in x-direction

28. Proposed system • Distribution of intensity error correlates roughly to the panning motion • Errors tends to be clustered in the temporal domain • Errors of homogeneous and texture regions are tend to randomly around zero

29. Proposed system • Sprite updating • Selecting frames with significant change in panning direction/speed 0 51 108 174 206

30. Proposed system • Sprite updating (cont.) Reconstruct next N frame from the sprite Compute the N error frames Blend the N error frames into a sprite-sized buffer(the sprite update frame) Encode and send the sprite update frameto the decoder MPEG4 I-VOP frame

31. Experimental results

32. Experimental results • Testing • Constructing sprite • Reconstructing frames from sprite • Compute PSNR • Comparison • Short-term motion estimation • Estimating between current and previous frame • Long-term motion estimation • Estimating between current frame and sprite • No parameters predicting • Long-term motion estimation by MPEG-4 VM • Long-term motion estimation by Smolic et. al.

33. Experimental results Short-term Long-term

34. Experimental results MPEG-4 VM Proposed method

35. Experimental results • PSNR MPEG-4 Short-term Long-term Proposed Smolic et. al.

36. Experimental results • Average PSNR (dB)

37. Experimental results • Selecting threshold Nr • Proposed method is better than simple short-term and long-term estimation Short-term 0.1 Long-term

38. Experimental results • Performance of sprite updating * Update frames is figured out from the major camera operations of the sequences

39. Conclusions

40. Conclusions • New global motion estimation method • Directly estimation from current frame to a chosen reference frame • Combing advantages of short-term and long-term estimation • Error accumulation prevented • Keeping reference frame close to current frame • Sprite updating • Encoding & sending sprite update frames • Errors of a group of reconstructed frames • Reducing sprite blurring