1 / 103

Block Loss Recovery Techniques for Image and Video Communications

Block Loss Recovery Techniques for Image and Video Communications. Jiho Park The Computational Intelligence Applications (CIA) Lab. Department of Electrical Engineering University of Washington May 21, 2002. Outline. Introduction Background for Block Recovery

vernon
Télécharger la présentation

Block Loss Recovery Techniques for Image and Video Communications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Block Loss Recovery Techniques for Image and Video Communications Jiho Park The Computational Intelligence Applications (CIA) Lab. Department of Electrical Engineering University of Washington May 21, 2002

  2. Outline • Introduction • Background for Block Recovery • Spatial Block Loss Recovery Technique • Temporal Block Loss Recovery Technique • Interpolation based Coding and Data Priority Setting • Conclusion & Future Work

  3. Image and Video Coding Standards • Adopt block based coding (e.g. JPEG, MPEG, H.261/3+) • Lossy coding techniques • Images are segmented into blocks • Predictive Coding, Discrete Cosine Transform and Quantization are adopted

  4. Background –Block Coding & Transmission Error • Block based coding technique Discrete Cosine Transform N x N block Q RLC Packet • image/video transmission system. Encoding Packetizing Network Depacketizing Decoding Image/ Video Source

  5. Transmission & Error • Packet error during transmission causes all data loss in a corresponding block. • During transmission, some packets are lost due to bit error, congestion of network, noise burst, or other reasons. Packet

  6. Background – Error Correction • Automatic Retransmission Request(ARQ) – Decoder sends retransmission requests to encoder when error happens. pro : error-free. cons : additional delays, unacceptable for real-time. example : TCP/IP – unreal-time internet file transfer protocol • Forward Error Correction(FEC) – Encoder inserts additional codes so that decoder can correct errors. pro : error resilient. con : additional bits example : 18-bit FEC code in H.261(video-conferencing standard) • Error Concealment/Block Loss Recovery – Post-processing to restore damaged blocks by decoder pro : no additional bits. con : complex decoder

  7. Background –Error Concealment / Block Loss Recovery • Spatial EC : Exploiting spatially neighboring image data of a missing block. • Temporal EC : Exploiting temporally adjacent frame of video. t-1 t

  8. Spatial Block Loss Recovery • Spatial block loss recovery is suitable for image coding data, (e,g, JPEG, Image based coding frame of MPEG/H.261/263+)

  9. Previous Research (Spatial EC) -Edge-based Average Interpolation (Ancis ’99) • Averaging the values of pixels in the same position of surrounding blocks. where wk = 0, 1, according to edges in surrounding blocks

  10. Previous Research (Spatial EC) - Interpolation Exploiting Interblock Correlation (Hemami ’95) • To get weight, W, the values of boundary pixels in a missing block are set to the value of adjacent known pixels. • Weight, W, are computed by solving the equations using Linear Least Squares Problem. (1~64) (1~30)

  11. Previous Research (Spatial EC) – DCT-based interpolation (Ziad, 2000) • Set Y2 to zero =>

  12. Previous Research – Other Algorithms • Maximally Smooth Image Recovery (Wang, ’93) - minimizing errors between a missing block and surrounding pixels • DCT Coefficients Block Recovery (Park, ’97) - minimizing errors between boundary pixels of a missing block and surrounding pixels • Block Recovery using POCS (Sun, ’95)

  13. Previous Research – Problem Formulation • Conventional Algorithms - 1) Not adaptive to local image structure (Ziad, Hemami,…) 2) ill-utilizing surrounding image structure (hemami,…) 3) no ability of recover certain frequency bands (Sun, Ziad, Wang, Park…)

  14. Block Loss Recovery – Research Goal • To design a spatial block loss recovery Techninque • More Reliable • Adaptive to Image Structure • No Retransmission • No Additional Bits for error correction

  15. Projections based Block Recovery – Motivation • Conventional Algorithms use information of all surrounding area. • Using only highly correlated area

  16. Projections onto Convex Sets –Alternating Projections • Alternating Projections is projecting between two or more convex sets iteratively. Converging to a common point

  17. Signal Restoration using Alternating Projections – Papoulis-Gerchberg Algorithm Original Signal FT Band Limiting. Convex Set, C1 Corrupted Signal FT IFT C2 Identical Middle. Convex Set, C2

  18. Projections based Block Recovery – Algorithm • 2 Steps • Pre Process : 1) Edge orientation detection 2) Surrounding vector extraction 3) Recovery vector extraction • Projections : 1) Projection operator P1 2) Projection operator P2 3) Projection operator P3

  19. Pre Process 1 –Edge Orientation Detection • Edge orientation in the surrounding area(S) of a missing block(M). In order to extend the geometric structure to the missing block. • Simple line masks at every i, j coordinate in surrounding area(S) of the missing block(M) for edge detection. Horizontal Line Mask Vertical Line Mask

  20. Pre Process 1 – Edge Orientation Detection • Responses of the line masks at window W : • Total magnitude of responses : • Th > Tv ; Horizontal line dominating area Th < Tv ; Vertical line dominating area

  21. Pre Process 2 – Surrounding Vectors • Surrounding Vectors, sk, are extracted from surrounding area of a missing block by N x N window. • Each vector has its own spatial and spectral characteristic. • The number of surrounding vectors, sk, is 8N.

  22. Pre Process 3 – Recovery Vector • Recovery vectors are extracted to restore missing pixels. • Two positions of recovery vectors are possible according to the edge orientation. • Recovery vectors consist of known pixels(white color) and missing pixels(gray color). • The number of recovery vectors, rk, is 2. Vertical line dominating area Horizontal line dominating area

  23. Projections based Block Recovery –Projection operator P1 • Recovery vectors, ri, for i = 1, 2 • Surrounding vectors, sj, for j = 1 ~ 8N • Surrounding vectors, S, form a convex hull in N2-dimensional space • Recovery vectors, R, are orthogonally projected onto the line defined by the closest surrounding vector, si, j : Projection Operator P1.

  24. Projections based Block Recovery –Projection operator P1 • Projection operator P1 : Convex hull (formed by surrounding vectors, containing information of local image structure)

  25. Projections based Block Recovery –Projection operator P1 • Surrounding vectors, sj, for j = 1 ~ 8N • Recovery vectors, ri, for i = 1, 2 • The closest vertex, sdi, from a recovery vector, ri. or equivalently in DCT domain, • P1 :

  26. Projections based Block Recovery –Projection operator P2 • Convex set C2 acts as an “identical middle”. • Projection operator P2 :

  27. Projections based Block Recovery – Projection operator P3 • Convex set C3 acts as a convex constraint between missing pixels and adjacent known pixels, (fN-1 fN). where, and is a N x N recovery vector in column vector form. fN-1 fN

  28. Projections based Block Recovery – Projection operator P3 • Projection operator P3 :

  29. Projections based Block Recovery –Iterative Algorithm • Missing pixels in recovery vectors are restored by iterative algorithm of alternating projections : • N x N windows moving : Vertical line dominating area Horizontal line dominating area

  30. Projections based Block Recovery - Summary Edge Orientation Detection Surrounding Vector Extraction Recovery Vector Extraction Projection Operator P1 Projection Operator P2 Projection Operator P3 Iteration=I? All pixels?

  31. Simulation Results – Test Data and Error • 512 x 512 “Lena”, “Masquerade”, “Peppers”, “Boat”, “Elaine”, “Couple” • 176 x 144 “Foreman” • 352 x 240 “Flower Garden” • 8 x 8 pixel block loss • 16 x 16 pixel block loss • 8 x 8 consecutive block losses • Peak Signal – Noise – Ratio

  32. Simulation Results –Lena, 8 x 8 block loss Original Image Test Image

  33. Simulation Results –Lena, 8 x 8 block loss Ancis, PSNR = 28.68 dB Hemami, PSNR = 31.86 dB

  34. Simulation Results –Lena, 8 x 8 block loss Ziad, PSNR = 31.57 dB Proposed, PSNR = 34.65 dB

  35. Simulation Results –Lena, 8 x 8 block loss Ancis PSNR = 28.68 dB Hemami PSNR = 31.86 dB Ziad PSNR = 31.57 dB Proposed PSNR = 34.65 dB

  36. Simulation Results – Each Step Lena 8 x 8 block loss (a) (b) (c)

  37. Simulation Results –Peppers, 8 x 8 block loss Original Image Test Image

  38. Simulation Results – Peppers, 8 x 8 block loss Ancis, PSNR = 27.92 dB Hemami, PSNR = 31.83 dB

  39. Simulation Results – Peppers, 8 x 8 block loss Ziad, PSNR = 32.76 dB Proposed, PSNR = 34.20 dB

  40. Simulation Results – PSNR (8 x 8)

  41. Simulation Results –Lena, 8 x one row block loss Original Image Test Image

  42. Simulation Results –Lena, 8 x one row block loss Hemami, PSNR = 26.86 dB Proposed, PSNR = 30.18 dB

  43. Simulation Results –Masquerade, 8 x one row block loss Original Image Test Image

  44. Simulation Results –Masquerade, 8 x one row block loss Hemami, PSNR = 23.10 dB Proposed, PSNR = 25.09 dB

  45. Simulation Results –Lena, 16 x 16 block loss Original Image Test Image

  46. Simulation Results –Lena, 16 x 16 block loss Ziad, PSNR = 28.75 dB Proposed, PSNR = 32.70 dB

  47. Simulation Results –Foreman, 16 x 16 block loss Original Image Test Image Ziad, PSNR = 25.65 dB Proposed, PSNR = 30.34 dB

  48. Simulation Results –Flower Garden, 16 x 16 block loss Original Image Test Image Ziad, PSNR = 20.40 dB Proposed, PSNR = 22.62 dB

  49. Simulation Results – PSNR (Row, 16 x 16)

  50. Simulation Results –Mean & Variance of Error (8 x 8)

More Related