Fully Scalable Multiview Wavelet Video Coding

1. Fully Scalable Multiview Wavelet Video Coding Yu Liu and King Ngi Ngan Department of Electronic Engineering The Chinese University of Hong Kong Email: yliu@ieee.org knngan@ee.cuhk.edu.hk ISCAS2009, Taipei, Taiwan, May 24-27, 2009 Fully Scalable Multiview Wavelet Video Coding

2. Introduction Proposed Method Experimental Results Conclusion Introduction Proposed Method Experimental Results Conclusion Outline Fully Scalable Multiview Wavelet Video Coding

3. Multiview Video • Captured simultaneously by several cameras from different viewpoints • Straightforward solution: simulcast coding for each view • H.264/AVC [1] or wavelet-based VIDWAV [2] • Inter-view statistical dependencies • Multiview video coding (MVC) Full scalability in multiview video coding • Quality, spatial, temporal and view dimensions • Scalable coding: H.264/AVC-based JSVM [3] and wavelet-based SVC [2] • Can achieve part of the requirement, but view scalability is missing • H.264/AVC-based JMVM [4] • Some degree of temporal and view scalability • But spatial or quality scalability is not supported Introduction RelatedWork Introduction Proposed Method Experimental Results Conclusion Introduction Fully Scalable Multiview Wavelet Video Coding

4. Schemes for wavelet-based MVC • Inherent scalability due to subband/wavelet decomposition • 2D wavelet transform in spatial domain • Motion compensated temporal filtering (MCTF) in temporal domain • Disparity compensated view filtering (DCVF) in view domain • But no view scalability [5] or limited view scalability [6] is supported • Due to global MCTF/DCVF selection on Group of GOP basis • An attempt to provide full view scalability by Garbas et al. [7] • But the coding efficiency is not satisfying Introduction Related Work Introduction Proposed Method Experimental Results Conclusion Related Work Fully Scalable Multiview Wavelet Video Coding

5. Wavelet-based MVC Frameworks • Different wavelet decomposition structures along temporal and view axes • (a) Simulcast scheme[2] • Only MCTF, but no DCVF • (b) Regular scheme[5] • Multilevel MCTF for thetemporal subbands • Only DCVF in the lowest temporal subbands • (c) Adaptive scheme[5,6] • MCTF and DCVF interleaved • global MCTF/DCVF selection on Group of GOP basis Introduction Related Work Introduction Proposed Method Experimental Results Conclusion Related Work Fully Scalable Multiview Wavelet Video Coding

6. Analysis of temporal and inter-view correlation [8] • Temporal prediction mode is dominant for all sequences • But sometimes inter-view prediction is more efficient than temporal prediction for a number of blocks Locally Adaptive Inter-View-Temporal Structure 2-D Pipeline-Based Lifting Scheme Introduction Proposed Method Experimental Results Conclusion Locally Adaptive Inter-View-Temporal Structure (a) Prediction modes with first order inter-view and temporal neighbor pictures, (b) Probabilities of prediction modes (T: temporal mode, V: inter-view mode) [8] on the macroblock level Fully Scalable Multiview Wavelet Video Coding

7. Locally adaptive inter-view-temporal decomposition • Adopts the locally adaptive inter-view-temporal correlation analysis based on the macroblock level • Instead of the global temporal and inter-view correlation analysis based on the whole picture • Problem: how to implement the lifting steps of the MCTF and DCVF on the macroblock level within the same framework • Solution: the weighting lifting [9] is employed for MCTF or DCVF Locally Adaptive Inter-View-Temporal Structure 2-D Pipeline-Based Lifting Scheme Introduction Proposed Method Experimental Results Conclusion Locally Adaptive Inter-View-Temporal Structure Fully Scalable Multiview Wavelet Video Coding

8. Locally adaptive inter-view-temporal wavelet decomposition structure • The number of levels of inter-view transform at each temporal subband is related to that of temporal transform, but not in excess of the maximum predetermined number of levels of inter-view transform. Locally Adaptive Inter-View-Temporal Structure 2-D Pipeline-Based Lifting Scheme Introduction Proposed Method Experimental Results Conclusion Locally Adaptive Inter-View-Temporal Structure Fully Scalable Multiview Wavelet Video Coding

9. Selection of optimal decomposition structure • Adaptive wavelet decomposition scheme on the basis of GoGOP (Group of GOP) • Demands the significant memory requirement and computational complexity • Suffers from the problem of temporal boundary effects across GOP • And thus its coding performance depends on the temporal GOP size • In order to solve those problems, a new 2-D pipeline-based lifting scheme is proposed • An extension of 1-D pipeline-based lifting scheme [10] • on the basis of macroblock level • It does not physically break the multiview sequence into GoGOP but processes it without intermission • At most 4x4 frames are involved for one-level transform Locally Adaptive Inter-View-Temporal Structure 2-D Pipeline-Based Lifting Scheme Introduction Proposed Method Experimental Results Conclusion 2-D Pipeline-Based Lifting Scheme Fully Scalable Multiview Wavelet Video Coding

10. Pipeline processing-based lifting schemes Locally Adaptive Inter-View-Temporal Structure 2-D Pipeline-Based Lifting Scheme Introduction Proposed Method Experimental Results Conclusion 2-D Pipeline-Based Lifting Scheme 2-D case of (c) forward and (d) inverse inter-view-temporal transform • 1-D case of • forward and • Inverse temporal transform Fully Scalable Multiview Wavelet Video Coding

11. Two multiview QVGA (320x240) sequences: (a) Ballroom and (b) Race1 • Eight views with frame rate of 25 Hz or 30 Hz, captured by a parallel camera array with 20 cm spacing • First 128 frames of each sequence are encoded to generate only one bitstream for one multiview video (including eight views) • PSNR is averaged over all views and bit rate is given per view Different temporal/inter-view wavelet decomposition schemes are compared: • Simulcast scheme • Regular scheme • Adaptive scheme • Proposed scheme Experimental Results Introduction Proposed Method Experimental Results Conclusion Experimental Results Fully Scalable Multiview Wavelet Video Coding

12. The performance comparison of different wavelet decomposition schemes for tested multiview sequences: Ballroom and Race1 Experimental Results Introduction Proposed Method Experimental Results Conclusion Experimental Results Fully Scalable Multiview Wavelet Video Coding

13. The performance comparison between the proposed scheme and simulcast scheme with different quality, temporal and view scalability for tested multiview sequences: Ballroom and Race1 Experimental Results Introduction Proposed Method Experimental Results Conclusion Experimental Results Fully Scalable Multiview Wavelet Video Coding

14. Coding Performance Comparison • The PSNR gain can be up to 1.18 dB and 0.88 db over the regular and adaptive schemes at low bit rates • Up to 2.49 dB coding gain is observed at low bit rates, compared with the simulcast scheme Notes: We do not demonstrate the coding results for spatial scalability. The reason is that all of these schemes naturally inherit this scalability from wavelet-based SVC. Experimental Results Introduction Proposed Method Experimental Results Conclusion Experimental Results Fully Scalable Multiview Wavelet Video Coding

15. Memory and Complexity Comparison • Simulcast scheme • the least requirement of memory and computation, due to the only temporal correlation exploitation • Regular scheme • Slightly increase the requirement of memory and computation than the simulcast, due to only exploiting the inter-view correlation at the lowest temporal subbands • Adaptive scheme • Significantly increase the memory and computational complexity, due to the selection on the basis of GoGOP • Proposed scheme • Moderate memory and complexity, due to the 2-D pipeline-based inter-view-temporal lifting on macroblock basis Experimental Results Introduction Proposed Method Experimental Results Conclusion Experimental Results Fully Scalable Multiview Wavelet Video Coding

16. Inter-View-Temporal Lifting-based Wavelet Coding Technique for Fully Scalable Multiview Video Coding • An important scalability feature • view scalability, besides other three scalabilities • 2-D pipeline-based lifting scheme • to remove both of temporal and view boundary effects • Local adaptive inter-view-temporal structure • to exploit local inter-view-temporal correlation on macroblock level Conclusion Introduction Proposed Method Experimental Results Conclusion Conclusion Fully Scalable Multiview Wavelet Video Coding

17. Thank You ! Q&A Fully Scalable Multiview Wavelet Video Coding