1 / 39

EE5359 Spring 2008 Comparative study of Motion Estimation (ME) Algorithms

EE5359 Spring 2008 Comparative study of Motion Estimation (ME) Algorithms. Khyati Mistry 1000552796 04/29/2008. Comparative study of Motion Estimation (ME) Algorithms. Objective:

yoshi
Télécharger la présentation

EE5359 Spring 2008 Comparative study of Motion Estimation (ME) Algorithms

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. EE5359Spring 2008Comparative study of Motion Estimation (ME) Algorithms Khyati Mistry 1000552796 04/29/2008

  2. Comparative study of Motion Estimation (ME) Algorithms Objective: • The goal of this project is to do a comparative study of the different motion estimation techniques and search algorithms that have been proposed for interframe coding in moving video sequences taking into consideration the different kinds of motion of the moving objects in successive frames –translational, rotational, zooming etc EE5359: Project Report

  3. Comparative study of Motion Estimation (ME) Algorithms • Introduction The motion estimation module will create a model for the current frame by modifying the reference frames such that it is a very close match to the current frame. This estimated current frame is then motion compensated and the compensated residual image is then encoded and transmitted. Fig 1: Motion estimation block diagram [1] EE5359: Project Report

  4. The objective is to estimate the motion vectors from two time sequential frames of the image. The objective is to estimate the motion vectors from two time sequential frames of the image. The motion of an object in the 3-d object space is translated into two successive frames in the image space at time instants t1 and t2 as shown in Fig 2. Where t1, t2 = represent the time axis such that t2 > t1 (X, Y) = Image-space coordinates of P in the scene at time t1 (X’, Y’) = Image space coordinates of P at time t2 > t1 (x, y, z) = Object-space coordinates at a point P in the scene at time t1 Comparative study of Motion Estimation (ME) Algorithms Fig 2:Basic geometry for motion estimation [2] EE5359: Project Report

  5. Comparative study of Motion Estimation (ME) Algorithms • Broad Classification of Motion Estimation Algorithms: Fig 3. Motion estimation algorithms [10] EE5359: Project Report

  6. Comparative study of Motion Estimation (ME) Algorithms • Frequency domain algorithms: In these algorithms, the algorithm is applied on the transformed coefficients [10]. The different algorithms/techniques used are phase correlation [21], matching in wavelet and DCT domain [17]. • Time domain algorithms: These comprise of matching algorithms and gradient-based algorithms [1]-[16]. • Block matching algorithms: match all or some pixels in the current block with the block in the search area of reference frame based on some cost function [1]-[15]. • Feature based algorithms: match the meta information/data of the current block with that of the block in reference frame [16]. EE5359: Project Report

  7. Comparative study of Motion Estimation (ME) Algorithms Time domain Vs frequency domain algorithms Fig 4: (a) Conventional video encoder with motion estimation and compensation in the spatial domain. (b) Simplified video encoder with motion estimation and compensation in the transform domain. [17] EE5359: Project Report

  8. Comparative study of Motion Estimation (ME) Algorithms • In the frequency domain, the phase correlation algorithm provides accurate predictions but is based on the fast Fourier transform (FFT), which is incompatible with current DCT-based video coding standards and which has high computational complexity since a large search window is necessary. Also, it gives poor performance when block size is small and motion is not pure translational. • DCT-based motion estimation simplifies the conventional DCT-based video coders achieving spatial redundancy reduction through DCT and temporal redundancy reduction through motion estimation and compensation. • In the conventional DCT-based video coder, the feedback loop has the following functional blocks: DCT, inverse DCT (IDCT), quantizer, inverse quantizer, and spatial domain motion estimation and compensation. • If motion estimation and compensation are performed entirely in the DCT domain, IDCT can be removed from the feedback loop. Therefore, the feedback loop contains the reduced functional blocks: quantizer, inverse quantizer, and transform domain motion estimation and compensation (Fig 4) • High computational complexity is still the main drawback of the DCT-based motion estimation and compensation approach. EE5359: Project Report

  9. Comparative study of Motion Estimation (ME) Algorithms Major Aspects of motion estimation techniques: • Reduced computational complexity • Good visual quality in terms of representation of true motion • High compression ratio EE5359: Project Report

  10. Comparative study of Motion Estimation (ME) Algorithms The computational complexity of a motion estimation technique can be determined by three factors • The search algorithm: decides the overall computational complexity and accuracy of motion estimation • Search area : the span of search window to find the best match • Cost function: the different cost metrics used to find the best match EE5359: Project Report

  11. Comparative study of Motion Estimation (ME) Algorithms Cost Functions The different cost functions that are used to find the best possible match are listed [10] • CCF (cross correlation function) CCF(dx,dy) = (MVx, MVy) = max (dx,dy)ЄR2CCF(dx,dy) EE5359: Project Report

  12. Comparative study of Motion Estimation (ME) Algorithms • MSE (mean square error) MSE(dx,dy) = (MVx, MVy) = min (dx,dy)ЄR2 MSE(dx,dy) EE5359: Project Report

  13. Comparative study of Motion Estimation (ME) Algorithms • MAE (mean absolute error): MAE(dx,dy) = (MVx, MVy) = min (dx,dy)ЄR2 MAE(dx,dy) EE5359: Project Report

  14. Comparative study of Motion Estimation (ME) Algorithms • SAD (sum of absolute differences) SAD(dx,dy) = (MVx, MVy) = min (dx,dy)ЄR2 SAD(dx,dy) EE5359: Project Report

  15. Motion estimation Algorithms 1. Full search This is the brute force technique in which each block is matched with the current block to find the best possible match. It is a very simple and accurate technique. Not efficient because it involves high computational cost. So more efficient algorithms were proposed but there was always a tradeoff between efficiency and image quality. Fig 5: Full search (Brute force technique) Comparative study of Motion Estimation (ME) Algorithms EE5359: Project Report

  16. Comparative study of Motion Estimation (ME) Algorithms 2.Fast motion estimation algorithms To meet the requirements of low power consumption for most of the battery powered real time visual communication applications and reduced computational complexity, new fast algorithms have been developed. They reduce the search area at the cost of visual quality; however the visual quality is not impaired beyond certain threshold. The various fast motion estimation algorithms can be classified as [10] -Search area subsampling a. fixed-search area: TSS,OTS,NTSS,4SS,OSA,Cross Search, 2DLOG b. adaptive search area: zone based search, spiral based search -Hierarchical/Multiresolution Subsampling, HPDS, Gaussian, reduced pel -Pel Decimation 2:1, 4:1, 16:1 -Prediction based Temporal, Spatial, Linear, Non-Linear -Feature Matching IPM, SEA, BBM, BFM, BPM EE5359: Project Report

  17. Three step search Three step search (TSS) [5], employs rectangular search patterns with different sizes. Koga et al [5] introduced this algorithm in 1981. It became very popular because of its simplicity and also robust and near optimal performance. It searches for the best motion vectors in a coarse to fine search pattern. One problem that occurs with the three step search is that it uses a uniformly allocated checking point pattern in the first step, which becomes inefficient for small motion estimation. Fig 6:TSS Procedure: Arrow denotes the direction of minimum distortion in each step [11] Comparative study of Motion Estimation (ME) Algorithms EE5359: Project Report

  18. Comparative study of Motion Estimation (ME) Algorithms • Experimental results [5] show that the block motion field of a real world image sequence is usually gentle, smooth and varies slowly. This results in center-biased global minimum distribution instead of uniform distribution. • So, the checking point pattern should also be center- biased. EE5359: Project Report

  19. Comparative study of Motion Estimation (ME) Algorithms New three step search (NTSS) • To remedy the problem with TSS,NTSS was proposed. • It employs a center-biased checking point pattern in the first step, which is derived by making the search adaptive to the motion vector distribution, and a halfway-stop technique to reduce the computation cost. • Fig 7-a [5] and Fig 7-b [5] explain the implementation. • In Fig 7-a, filled circles are the checking points in the first step of TSS, squares are the 8 extra points added in the first step of NTSS, and triangles explain how the second step search is performed if the minimum BDM (block distortion measure) in the first step is at one of the 8 neighbors of the window center. • Fig 7-b shows the block diagram of NTSS algorithm. Fig 7: Implementation details of NTSS Algorithm [5] EE5359: Project Report

  20. Four step search This algorithm also exploits the center-biased property in its search. The computational complexity of the four-step search is less than that of the new three step search, while the performance in terms of quality is as good. It is also more robust than the three step search and it maintains its performance for image sequences with complex movements like camera zooming and fast motion. Hence it is a very attractive strategy for motion estimation. Fig. 8: Search patterns of the 4SS (a) First step, (b) second/third step (c)second/third step (d) fourth step [13] Comparative study of Motion Estimation (ME) Algorithms EE5359: Project Report

  21. Comparative study of Motion Estimation (ME) Algorithms • Search points needed (considering the displacement parameter p in [-7,7] range)- Minimum: 9+8 = 17 Worst case: 9+5+5+8= 27 (Large motion) • Thus computational complexity is just two block matches more compared to TSS and lesser compared to 33 block matches in NTSS. • Performance comparable to NTSS. Fig. 9: EE5359: Project Report

  22. 2-D Logarithmic search Although 2-D logarithmic algorithms require more steps than the three step search, it can be more accurate, especially when the search window is large[6]. This procedure requires onlysearching 13to 21 locations. Displacement parameter p = 5 Fig 10: 2-D Logarithmic search [6] Comparative study of Motion Estimation (ME) Algorithms So n = 2 for p=5 EE5359: Project Report

  23. One-at-a-time search (OTS) One-at-a-time search (OTS) is a basic two-variable optimization search method. Each variable is adjusted with the other fixed. Direction of search in each step will be parallel to one of the coordinate axes. Fig. 11 [11]illustrates the OTS for a convex function in two variables. Fig 11: Illustration of one at a time search [11] Comparative study of Motion Estimation (ME) Algorithms EE5359: Project Report

  24. Conjugate direction search (CDS) An extension of the one-at-a-time search is the conjugate direction search (CDS) as shown in Fig 12 [11]. The directions of search, rather than being kept fixed, are changed to improve the search efficiency. The CDS obtains n linearly independent conjugate direction vectors, n being the number of variables in the objective function. A succession of one-at-a-time searches in each of n sets of independent directions starting with the coordinate directions is carried out. Fig 12: Illustration of conjugate direction search [11] Comparative study of Motion Estimation (ME) Algorithms EE5359: Project Report

  25. Comparative study of Motion Estimation (ME) Algorithms • Diamond search (DS) • As indicated in Table 1, about 52.76% to 96.09% of the motion vectors are enclosed in a circular support with a radium of 2 pels and centered on the zero-motion position. • TSS uses a sparse searching point pattern. This could yield the search path in wrong direction and thus give a wrong optimum point. • Other methods which use a smaller search pattern can cause the MDB to be trapped to a local minimum. Table 1: Motion vector distribution aggregately measured at various motion distances (in pel) with regard to the center position using the Full Search algorithm based on MSE matching criterion. [15] EE5359: Project Report

  26. Fig 14: An appropriate search pattern support—circular area with a radium of 2 pels. The 13 crosses show all possible checking points within the circle [15] Fig 15: Two search patterns derived from Fig 14 are employed in the proposed DS algorithm [15] Comparative study of Motion Estimation (ME) Algorithms • Diamond search (DS) (contd..) EE5359: Project Report

  27. Comparative study of Motion Estimation (ME) Algorithms • The DS algorithm employs two search patterns as illustrated in Fig. 15, which are derived from the crosses (×) in Fig. 14. • The first pattern, called large diamond search pattern (LDSP), comprises nine checking points from which eight points surround the center one to compose a diamond shape. • The second pattern consisting of five checking points forms a smaller diamond shape, called small diamond search pattern (SDSP). • In the searching procedure of the DS algorithm, LDSP is repeatedly used until the step in which the minimum block distortion (MBD) occurs at the center point. • The search pattern is then switched from LDSP to SDSP as reaching to the final search stage. • Among the five checking points in SDSP, the position yielding the MBD provides the motion vector of the best matching block. EE5359: Project Report

  28. Comparative study of Motion Estimation (ME) Algorithms Fig. 17: Diamondsearch path example which leads to the motion vector (-4, -2) in five search steps. There are 24 search points in total - taking 9, 5, 3, 3 and 4 search points in each step sequentially. [15] EE5359: Project Report

  29. Feature-matching Feature-matching algorithms can be seen as matching in meta information extracted from the current block and search area pels as shown in Fig.18 [10] In most of the feature matching algorithms, the meta information has a higher entropy than the original information, thus reducing the arithmetic computational load, as well as the memory bandwidth within the matching process. The feature extraction is performed by morphological filters, reduced pel resolution or projection methods. Integral projection matching (IPM), projection three step search (PTSS), successive elimination algorithm (SEA), binary block matching (BBM), bit plane matching (BPM), block feature matching (BFM) are some of the algorithms based on feature matching. Fig 18: Basic principle of feature-matching algorithms [10] Feature domain Previous Frame Search area subsampling And Feature matching process in the feature domain Feature extraction Current Frame Feature extraction High memory bandwidth Reduced memory bandwidth Comparative study of Motion Estimation (ME) Algorithms EE5359: Project Report

  30. Comparative study of Motion Estimation (ME) Algorithms 3. Fixed size block motion estimation (FSBME) and variable size block motion estimation (VSBME) [3] • The advantage of the former is that there is no need to provide segmentation information. However it is not very efficient when we have images that have both really fast and slow motions. • In this case the latter method is more useful because we can segment the image into smaller blocks in areas where there is complex motion and larger blocks when there is very little motion. EE5359: Project Report

  31. Comparative study of Motion Estimation (ME) Algorithms 3. Simulation • The simulation results for exhaustive search, TSS, NTSS, 4SS, DS on different test sequences have been tabulated. • Cronkite (256X256), toy vehicle (512X512), chemical plant (256X256) and caltrain test sequences [19] were used in the simulation. These are obtained from the digitization of 24 frames per second motion pictures. • The source code for the various algorithms is available from Matlab central file exchange [20]. This code was modified for different images. • The distance criterion method used is MAE. • Block size of 8X8 (16X16) is used. Search window size is (p + p + block size) X (p + p + block size) where p (=7) is the displacement parameter. • The various algorithms are compared based on their computational complexity and PSNR. EE5359: Project Report

  32. Comparative study of Motion Estimation (ME) Algorithms Table 2: Average number of computations needed for different ME algorithms EE5359: Project Report

  33. Comparative study of Motion Estimation (ME) Algorithms Table 3: Average PSNR (dB) for different ME algorithms EE5359: Project Report

  34. Comparative study of Motion Estimation (ME) Algorithms • Simulation results show that ES gives the best performance (highest PSNR) at the cost of high computational complexity. Motion estimation constitutes about 60% of the total computational load in encoders, so exhaustive search algorithm is not the most feasible technique to employ motion estimation. • TSS algorithm gives good performance but it can not always detect small motions. • NTSS is good for slow motions but its computational complexity is higher than that of TSS for large motion. • 4SS has a small initial step and is good to detect small motion. Its computational complexity is comparable to that of TSS and lower than that of NTSS. • DS performs well both computationally and in terms of quality. EE5359: Project Report

  35. Comparative study of Motion Estimation (ME) Algorithms References: • [1] Z. Wei et al “Efficient fast motion estimation algorithm for H.264 based on polynomial model,” IMACS Multiconference on computational engineering in System Applications, pp. 1654-1657, Oct. 4-6, 2006.  • [2] T. Huang, Y. Hsu and R. Tsai “Interframe coding with general two-dimensional motion compensation”, IEEE International Conference on ICASSP ’82 (Acoustics, Speech and Signal Processing), vol. 7, pp. 464 - 466, May 1982 • [3] M. Manikandan, P. Vijayakumar and N. Ramadass, “Motion estimation method for video compression - an overview”, Wireless and Optical Communications Networks, IFIP Intl’ Conference, pp. 5-11, 13 April 2006.  • [4] H. Ferhatosmanoglu et al, “Approximate nearest neighbor searching in multimedia databases”,Data Engineering, 2001. Proceedings. 17th International Conference on, pp. 503 – 511, 2-6 April 2001.  • [5] R. Li, B. Zeng, and M. L. Liou, “A new three-step search algorithm for block motion estimation” IEEE Transactions on Circuits and Systems Video Technology,vol. 4, pp. 438–443, Aug. 1994.  • [6] J. R. Jain and A. K. Jain, “Displacement measurement and its application in interframe image coding” IEEE Transactions on Communications, vol. 29, pp. 1799-1808, Dec. 1981. EE5359: Project Report

  36. Comparative study of Motion Estimation (ME) Algorithms • [7] J. Kim and S. Yang, “An efficient search algorithm for block motion estimation” IEEE Workshop on Signal Processing Systems, pp. 100-109, Oct. 1999. • [8] C. Zhu et al., “Enhanced hexagonal search for fast block motion estimation”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 14, pp. 1210 - 1214, Oct. 2004. • [9] E. Fernandez et al., “Reducing motion estimation complexity in MPEG-2 to H.264 transcoding”, IEEE International Conference on Multimedia and Expo, 2007 ,pp. 440-443, 2-5 July 2007. • [10] K. Peter, “Algorithms,complexity analysis and vlsi architectures for MPEG-4 motion estimation”, Kluwer Academic Publishers, Boston, 1999. • [11] R. Srinivasan and K. R. Rao, “Predictive coding based on efficient motion estimation” IEEE Transactions on Communications , vol. 33, pp. 888-896, Aug. 1985. • [12] I. Amre et al, “An efficient variable block size selection scheme for the H.264 motion estimation” 6th International Workshop on System-On-Chip for Real-Time application, pp. 5-9, Dec. 2006. EE5359: Project Report

  37. Comparative study of Motion Estimation (ME) Algorithms • [13] P. Lai-Man and M. Wing-chung, “A novel four-step search algorithm for fast blockmatching”, IEEE Transactions On Circuits and Systems for Video Technology, vol. 6, pp. 313-317, Jun. 1996. • [14]H. Zhongli and M.L. Lieo, “A high performance fast search algorithm for block matching motion estimation”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 7, pp. 826-828, Oct. 1997. • [15] S. Zhu and K. K. Ma “A new diamond search algorithm for fast block-matching motion estimation”, IEEE Transactions on Image Processing, vol. 9, pp. 287-290, Feb. 2000. • [16] Y. Chan and W. Siu , “An efficient search strategy for block motion estimation using image features”, IEEE Transactions on Image Processing, vol. 10, pp. 1223-1238, Aug. 2001. • [17] J. Lee et al, “An efficient architecture for motion estimation and compensation in the transform domain”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 16, pp. 191-201,Feb. 2006. • [18] C. Cheung and L. Po, “A hierarchical block motion estimation algorithm using partial distortion measure”, Proceedings International Conference on Image Processing, vol. 3, pp. 606-609, Oct. 1997 • [19] Test sequences obtained from http://sipi.usc.edu/database/database.cgi?volume=sequences • [20] Motion estimation algorithm source code available at Matlab central file exchange ‘ www.mathworks.com/matlabcentral/fileexchange’ • [21] “Phase correlation motion estimation” report available at http://visilab.unime.it/~ianni/slides_CV/liang_report-phase-correlation-motion-prediction.pdf EE5359: Project Report

  38. Comparative study of Motion Estimation (ME) Algorithms APPENDIX A ACRONYMS • 2DLOG 2-d Logarithmic Search • BBM Binary Block Matching • BDM Block Distortion Measure • BFM Block Feature Matching • BPM Bit Plane Matching • DCT Discrete Cosine Transform • DS Diamond Search • FS Full Search • FSBME Fixed Size Block Motion Estimation • H.264/AVC Video Coding Standard • HPDS Hierarchical Partial Distortion Search • IPM Integral Projection Matching • MAE Mean Absolute Error • MB Macro Block • ME Motion Estimation • MSE Mean Square Error • MV Motion Vector • NN Nearest Neighbor • NTSS New Three Step Search • OTS One-at-a-Time Search • PDC Pixel Difference Count • PSNR Peak Signal to Noise Ratio • SEA Successive Elimination Algorithm • TSS Three Step Search • VSBME Variable Size Block Motion Estimation EE5359: Project Report

  39. Comparative study of Motion Estimation (ME) Algorithms Questions? EE5359: Project Report

More Related