Source-Channel Prediction in Error Resilient Video Coding

1. Source-Channel Prediction in Error Resilient Video Coding Hua Yang and Kenneth Rose Signal Compression Laboratory ECE Department University of California, Santa Barbara

2. Outline • Introduction • Source-channel prediction • Simulation results • Conclusions ICME 2003

3. Introduction • Existent error resilient approaches on the prediction mechanism • Slice coding • limit prediction within certain non-overlapping spatial regions • Video redundancy coding • Multiple independently predicted “threads” • Multi-frame motion compensation • Multiple reference frames for prediction • Feature in common: assume the same underlying conventional prediction framework • Framework: separate source-channel coding • Prediction: past encoder reconstructed frames • Motion estimation criterion: minimum prediction error ICME 2003

4. Introduction • Via considering packet loss effects during encoding, joint source-channel coding usually achieves better error resilience than that of separate coding. • Our proposed approach • Prediction is based on expected decoder reconstruction of the previous frames. • Novelty Unlike all the other existent error resilient prediction schemes and all the other existent source-channel coding schemes, our proposed method is actually a source-channel prediction scheme. ICME 2003

5. Introduction p1 p2 p3 0 pi packet loss rate of video packet i. 1-p1 1-p2 1-p3 Encoder reconstruction, i.e. “best possible” decoder reconstruction: quantization loss only. p3 p1 p2 1-p3 p3 1-p2 Other possible decoder reconstructions: different transmission loss patterns. 1-p3 p2 p3 1-p3 p3 Expected decoder reconstruction: quantization loss & transmission loss. ICME 2003

6. unknown Random variable Introduction • Expected decoder reconstruction • Encoder’s estimate of the decoder reconstruction. • Given the packet loss rate, it can be accurately computed with the ROPE method. • Recursive optimal per-pixel estimate (ROPE) • Basic idea: • ROPE accurately computes these unknown quantities in a recursive manner for all the pixels of every frame. • Accurate & Low complexity • Frequently used to estimate end-to-end distortion in various RD optimization scenarios. Now we use these expectations for source-channel prediction. ICME 2003

7. Source-channel Prediction • Conventional prediction Source-channel prediction Prediction residue For pixel i in frame n: Original and predicted values Encoder and decoder reconstruction values of pixel j in frame n-1 to predict pixel i in frame n. Prediction error to be quantized ICME 2003

8. Plug in: Constant value: Not the actual predictor of the decoder Criterion I Source-channel Prediction • Source-channel prediction is the optimal prediction in the sense of minimum MSE end-to-end distortion. • Pending problem: motion estimation criterion ? Criterion in the conventional scheme ICME 2003

9. Criterion II Random variable: Actual predictor of the decoder Source-channel Prediction • Pending problem: motion estimation criterion? (cont.) • Criterion II is superior than Criterion I in that it explicitly accounts for the randomness of the decoder’s actual predictor. ICME 2003

10. Source-channel Prediction • Another interpretation of Criterion II While Criterion II considers the properly weighted impacts of both DR and DD , in contrast, Criterion I only considers DR . In this sense, Criterion II is more “comprehensive”. ICME 2003

11. Simulation Results • Simulation conditions • H.263+ video codec • System performance: average luminance PSNR • 50 different packet loss patterns • Testing scenarios • No INTRA Updating • Periodic INTRA Updating For packet loss rate p, coding a MB in INTRA mode once for every 1/p frames. • R-D optimized INTRA Updating For each MB, select its coding mode as INTER or INTRA with the R-D criterion. ICME 2003

12. (a) No INTRA updating ( p = 10%) (b) Periodic INTRA updating. (c) RD optimal INTRA updating. ICME 2003

13. Simulation Results • Observations • The proposed “SCP_CII” method consistently offers the best performance, which proves our previous analysis. • When INTRA updating is more effectively performed, smaller gains are achieved by “SCP_CII” over “EP”. Hence, the gain depends on how much damage of packet loss is not accounted for in the conventional scheme. • Similar results also hold for other testing sequences, e.g., carphone, miss_am, salesman, etc. ICME 2003

14. Demo Conventional prediction based on encoder reconstruction (PSNR = 25.06dB) Source-channel prediction based on expected decoder reconstruction (PSNR = 26.72dB) Foreman, QCIF, 30f/s, 300kb/s, packet loss rate = 10%, periodic Intra update. ICME 2003

15. Conclusions • Novelty: the proposal of further enhancement of error resilience via fundamental modification of the conventional prediction structure. • Source-channel prediction based on expected decoder reconstruction, which uses ROPE to get accurate estimate of decoder quantities. • In spite of the loss in source coding gain due to the lower source prediction quality, our scheme achieves better overall R-D tradeoff than the conventional scheme. • We identify the subtle points in selecting the motion estimation criterion, and shows that it is advantageous to use the criterion of minimizing the expected prediction error. ICME 2003

16. Thanks! ICME 2003