Chen-Yu Tseng ( 曾禎宇 ) Department of Electronic Engineering, National Chiao Tung University,

1. Error Resilience & Error Concealment Based on Motion Vector Correction & Missing Frame Prediction Chen-Yu Tseng (曾禎宇) Department of Electronic Engineering, National Chiao Tung University, Taiwan, R.O.C.

2. Introduction Transmitted Video Wireless Network Error Resilience Received Video Packet loss Error Concealment

3. Outline • Compressed Video Over Wireless Network • Prior Arts of Error Concealment • Proposed Error Concealment • Motion Vector Correction • Motion Recovery • Proposed Error Resilience Based on Motion Vector Correction • Experiments • Conclusion

4. Compressed Video Over Wireless Network Packet-Based Video Transmission Encoded Sequence Decoded Sequence P P I I Packet Loss P P P P P P P P Error Propagation time time

5. Prior Arts of Error Concealment • Block-Level Concealment • Spatial Based • Spatial-Temporal Based • Motion Vector Field Based • Frame-Level Concealment Missing Block Correctly Received Parts

6. Block-Level Concealment • Spatial Interpolation Based [9-11] Original Sequence Edge Preserving Bilinear Interpolation

7. Block-Level Concealment • Spatial-Temporal Based [12-15] Missing MB Substitute MB Reference Frame Current Frame Boundary Matching Algorithm (BMA) W. M. Lam et al. [12]

8. Block-Level Concealment • Motion Vector Field Based [16-17] Recovered MV Missing MB Substitute MB Reference Frame Current Frame Interpolation in MV Field

9. Whole-Frame Loss Example of 3G Applications: • QCIF Video Transmission at 64 kbps. Frame Rate: 10 fps. Average 800 bytes per frame • General Packet Size: 1kB Packet Loss May Cause Whole-Frame Loss!!

10. Frame-Level Concealment • Motion Vector Recovery Based on Optical Flow Recovered MV Ref MV Substitute MB Reference Frame Current Missing Frame

11. Frame-Level Concealment • S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “An error concealment algorithm for streaming video,” Proc. ICIP 2003, 2003 [22] • S. Belfiore, M. Grangetto, E. Magli, G. Olmo, “Concealment of whole-frame loss for wireless low bit-rate video based on multiframe optical flow estimation,” IEEE Trans. Multimedia, vol. 7, no. 2, pp. 316-329, Apr. 2005. [23] • P. Baccicht, D. Bagni, A. Chimienti, L.Pezzoni, and F. Rovati, “Frame concealment for H.264/AVC decoders,” IEEE Trans. Consumer Electronics, vlo. 51, no. 1, pp. 227-233, Feb. 2005. [24] • Zhenyu Wu and J. M. Boyce, “An error concealment scheme for entire frame losses based on H.264/AVC,” Proc. ISCAS 2006, May 2006.[25]

12. Optical Flow (OF) • Motion Consistency t-2 t-2 t-1 t-1 t time Optical Flow t

13. Motion Recovery Based on Optical Flow Packet lost t-2 time time t-1 Corrupted Optical Flow Original Optical Flow t time Optical Flow

14. Optical Flow Estimation • General Model of Optical Flow Equation[27] • Motion Vector substitutes for OF (1) (2) (i, j) (iF, jF) Fn-1 Fn Motion Vector Projection

15. MV Versus OF • MV  Block Matching • OF  Motion Trajectory Block Matching Reference Frame Foreman #4 Current Frame Foreman #5

16. Temporal Aliasing Problem Wrong MV Correct MV Projected MV Projected MV t-2 t-2 t-1 t-1 t Missing Frame t Correct MV Projection Wrong MV Projection

17. Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels

18. Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels

19. Pixel-based Concealment • Estimation of optical flow with temporal regularization n n-2 n-1 MV MV MV MV TERM MVH MVH TERM MVH MVH There are other temporal interpolators for the MVH, including median filter and weighted averages decaying overtime. Belfiore et al. [22] find that mean value consistently yields the best result, thus validating the linear velocity assumption.

20. Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels t t-1

21. Pixel-based Concealment • Schematic of Belfiore’s Algorithm[22] Reference Frame Buffer Estimation of optical flow with temporal regularization Spatial regularization of FMV field Projection of prev. frame onto missing frame Filtering and downsampling Interpolation of missing pixels t t-1

22. Pixel-based vs. Block-Based t t t-1 t-1 Pixel-based Block-based

23. Reference frame buffer Searching of suitable reference Motion vector projection Statistics collection MB-Level MV field estimation Block-Level MV field estimation Picture reconstruction t t-1 Block-based Concealment • Schematic of P. Baccicht’s Algorithm[24]

24. t Problems of MV Projection • Conflict State • Non-covered State Non-covered State Conflict State t-1

25. Problems of MV Projection • Conflict State • Non-covered State Possible Reasons • Wrong Motion Vector • Object Warping, Occlusion, or Non-translation Motion. • Frame Boundary

26. Outline • Compressed Video Over Wireless Network • Prior Arts of Error Concealment • Proposed Error Concealment • Motion Vector Correction • Motion Recovery • Proposed Error Resilience Based on Motion Vector Correction • Experiments • Conclusion

27. Motion Vector Correction • Motion Vector  True Motion Trajectory • Property of True Motion Trajectory • Temporal Consistency • Spatial Consistency Temporal Consistency t-2 t-1 t Spatial Consistency

28. Motion Vector CorrectionBased on Fuzzy Logic Motion Vector Field Original MV: Corrected MV: Reliability of : Spatial Reliability: Temporal Reliability: Spatial Difference Temporal Difference

29. Temporal Reliability • Temporal Difference t-2 t-2 t-1 t-1 t t time time High temporal difference Low temporal reliability Low temporal difference High temporal reliability 1 0

30. 1 0 Spatial Reliability • Spatial Difference Motion Vector Field Motion Vector Field

31. Motion Vector CorrectionBased on Fuzzy Logic y x Motion Vector Field Original Frame Original MV Field Corrected MV Field

32. MV Projection with MV Correction y x Original MV Field Corrected MV Field t t t-1 t-1

33. Proposed MV Projection • Why Concealment? Release Error Propagation Can We Skip from Error Frame? P P I I P P P P P P P P Error Propagation Error Frame Error Frame time time

34. MV Projection Forward vs. Backward Forward MV Projection Packet lost Packet lost Backward MV Projection time time time Corrupted Optical Flow Original Optical Flow Corrupted Optical Flow

35. MV Projection Forward vs. Backward OF OF Reference Frame Reference Frame Missing Frame Missing Frame Backward projected OF Forward projected OF OF (a) Forward motion projection (b)Backward motion projection

36. MV Projection Forward vs. Backward Backward Projected MV Missing frame Missing frame (i, j) (iF, jF) (iB, jB) Fn-1 Fn (i, j) Fn-1 Fn Fn+1

37. Advantage ofBackward MV Projection • Easy to implement. • Avoid from conflict or non-cover states. • Invariance of I-frame position. Missing frame Missing frame P P I I P P P time P P time P P P Missing frame No MV Missing frame P P P P I I P time P P time P P P Forward Motion Projection Backward Motion Projection

38. Outline • Compressed Video Over Wireless Network • Prior Arts of Error Concealment • Proposed Error Concealment • Motion Vector Correction • Motion Recovery • Proposed Error Resilience Based on Motion Vector Correction • Experiments • Conclusion

39. Schematic of Proposed System Wireless Network Encoder with MV Correction Decoder with Missing Frame Prediction Error Resilience Error Concealment

40. Motion Vector Correction • Spatial-Temporal MV Correction (3-D) Iterative Correction time … n-2 MV Field n x n-1 The number of MV correction iteration effects the result of concealment. … y The PSNR is the concealment result (foreman FLR10) with different number of MV correction iteration

41. …. …. …. …. …. …. MVn-2 MVn-2 MVn-2 MVn-2 MVn-2 MVn-2 MVn-1 MVn-1 MVn-1 MVn-1 MVn-1 MVn-1 MVn MVn MVn MVn MVn MVn MVn+1 MVn+1 MVn+1 MVn+1 MVn+1 MVn+1 MVn+2 MVn+2 MVn+2 MVn+2 MVn+2 MVn+2 …. …. …. …. …. …. Proposed 2-StepMotion Vector Correction Forward MV correction Backward MV correction Iteration time

42. Concealment PSNRwith Different MV Correction The PSNR is the concealment result (foreman FLR10) 2-Step: 33.4666 dB Bidirectional: 33.1826 dB

43. Motion Compensation PSNR PSNR Drop With MV Correction : 32.0382 dB Without MV Correction: 34.5538dB (foreman)

44. PSNR Drop Effect: Increase the bit rate Reason: Inconsistent Motion MV correction is based on motion consistency. In this situation, concealment is also hard to work. We preserve the original MV to avoid MC PSNR drop.

45. PSNR Drop Control Factor • If Residual_MADcorrect > • f * Residual_MADuncorrect • MVcorrect = MVoriginal • Residualcorrect = Residualoriginal ME MC MV Correction MC f is the control factor

46. MC_PSNR vs. EC_PSNR Foreman QCIF MC_PSNR: Motion Compensation PSNR EC_PSNR: Error Concealment PSNR (Backward MV concealment FLR10)

47. MC_PSNR vs. EC_PSNR Foreman QCIF MC_PSNR: Motion Compensation PSNR EC_PSNR: Error Concealment PSNR (Backward MV concealment FLR10)

48. Error Concealment Results • Foreman QCIF Backward Concealment FLR10 (#63) MAD Factor = inf MAD Factor = 2.75

49. Refinement of Concealment • MV Temporal Consistency Check t-2 t • Frame Spatial Consistency Check

50. MV Temporal Consistency Check t-2 t Difft(i, j) = || MVt(i, j) – MVt-2(iB, jB) || ,Where (iB, jB) = (i, j) + 2*MVt(i, j) If Difft(i, j) > Diff_temporal_TH MVt(i, j) = MV’t(i, j) , if Diff_v(i, j) < Diff_h(i, j), MV’t(i, j) = ( MVt(i-1, j) + MVt(i+1, j) )/2 else MV’t(i, j) = ( MVt(i, j-1) + MVt(i, j+1) )/2 where Diff_v(i, j) = || MVt(i-1, j) - MVt(i+1, j) || and Diff_h(i, j) = || MVt(i, j-1) - MVt(i, j+1) || MV Refinement