1 / 35

Multiview Video

Multiview Video. Kai-Chao Yang. Outline. History of Video Coding Standards Definition of Multiview Video Applications of Multiview Video Concept of Stereo Video Multiview TV/Video System Multiview Content Capture Correction Coding Display. History of Video Coding Standards. H.261.

vesna
Télécharger la présentation

Multiview Video

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Multiview Video Kai-Chao Yang VCLAB, National Tsing Hua University, Taiwan

  2. Outline • History of Video Coding Standards • Definition of Multiview Video • Applications of Multiview Video • Concept of Stereo Video • Multiview TV/Video System • Multiview Content • Capture • Correction • Coding • Display VCLAB, National Tsing Hua University, Taiwan

  3. History of Video Coding Standards H.261 H.263 H.263+ H.263++ ITU-T VCEG MPEG-1 MPEG-4 v2/visual ISO MPEG H.262/MPEG-2 H.264/MPEG4 AVC FRExt SVC MVC JVT 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 VCLAB, National Tsing Hua University, Taiwan

  4. Definition of Multiview Video • Multiview video • Multiple cameras are used to simultaneously acquire various viewpoints of a scene. • Multiview video coding • Encoding of sequences captured simultaneously from multiple cameras using a single video stream. VCLAB, National Tsing Hua University, Taiwan

  5. Applications of Multiview Videos • Free viewpoint TV • The viewers can experience the free viewpoint navigation within the range covered by the cameras • http://www.youtube.com/watch?v=0yP_J6M4fiU • Three-dimensional TV • Immersive teleconference VCLAB, National Tsing Hua University, Taiwan

  6. Examples (1/4) • Entertainments • 3D game, FVV, … • Medicine • 3D microscope, 3D endoscope, … • Education • 3D model, 3D classroom, … • Surveillance • Nurse, object recognition and tracking, … • Simulation • Flight simulation, … VCLAB, National Tsing Hua University, Taiwan

  7. Examples (2/4) • 3D microscope and 3D endoscope http://www.njneurosurgeons.com/news/2010/04/19/new-jerseys-first-3d-endoscopic-surgery-performed-by-ninj-faculty.html VCLAB, National Tsing Hua University, Taiwan

  8. Examples (3/4) • Pedestrian Tracking Kyungnam Kim and Larry S. Davis, "Multi-Camera Tracking and Segmentation of Occluded People on Ground Plane using Search-Guided Particle Filtering", European Conference on Computer Vision (ECCV), LNCS, 2006. VCLAB, National Tsing Hua University, Taiwan

  9. Examples (4/4) • Flight Simulation VCLAB, National Tsing Hua University, Taiwan

  10. Concept of Stereo Image/Video 3D information: Determination of relative depths in a perceived scene Stereopsis Accommodation of the eyeball Occlusion Linear perspective Vertical position … 2D knowledge VCLAB, National Tsing Hua University, Taiwan

  11. Concept of Stereo Video (3D Video) • Stereopsis • visual perception leading to the sensation of depth from the two slightly different projections of the world onto the retinas of the two eyes VCLAB, National Tsing Hua University, Taiwan

  12. Concept of Stereo Video VCLAB, National Tsing Hua University, Taiwan

  13. Multiview TV System Content Encoding Delivery Decoding Display VCLAB, National Tsing Hua University, Taiwan

  14. Multiview Content VCLAB, National Tsing Hua University, Taiwan

  15. Multiview Content • Camera capture • Stereo cameras • Camera array • Computer generated • Objects is defined • Depth is inherent • Conversion from 2D video • Detecting objects • Assigning depth • Filling occluded parts VCLAB, National Tsing Hua University, Taiwan

  16. Camera Capture • Stanford multi-camera array (128 video cameras) • Google street view camera (Image only) • Panasonic Full-HD 3D camera VCLAB, National Tsing Hua University, Taiwan

  17. Correction • Rectification of misalignment • Normalization of colors VCLAB, National Tsing Hua University, Taiwan

  18. Multiview Video Coding VCLAB, National Tsing Hua University, Taiwan

  19. Representation Formats (Full-Resolution) • Full-Resolution Stereo and Multiview • Double data rate for stereo videos • N-fold data rate for N-view videos • Efficient compression is the key issue • MVC extension of H.264/AVC VCLAB, National Tsing Hua University, Taiwan

  20. L R L R Representation Formats (Stereo Interleaving) • Stereo Interleaving • A multiplex of the two views into a single sequence • Spatial multiplexing • The left and right views are sub-sampled and interleaved into a single frame • Temporal multiplexing • The left and right views are interleaved as alternating frames • Advantages • Compatible with existing codecs and delivery infrastructure • Drawbacks • Loss of spatial or temporal resolution VCLAB, National Tsing Hua University, Taiwan

  21. Representation Formats (Depth-based) • Depth-based Format • 2D video + depth map • Advantages • Backward compatibility with the older coding standards • Supporting both stereo and multiview displays • Depth is adjustable • Drawbacks • Limited depth range • Occlusions • Other views have to be synthesized Synthesized view 2D view VCLAB, National Tsing Hua University, Taiwan

  22. Multiview Video Coding • Single view • Multi-view • Independent encoding • Low coding complexity but also low coding efficiency • Inter-view prediction • Exploiting both spatial and temporal redundancy I B B B P B B B P I B B B P B B B P I B B B P B B B P VCLAB, National Tsing Hua University, Taiwan

  23. I B P P P P P P P P P P B B B B B B B B B B B B B B B B B B P B P B B B B B B B B B P P P P P P P P P B B B B B B B B B P B P B B B B B B B B B B B B B B B B B B B I B P I B P I P Multiview Video Coding • Multiview camera VCLAB, National Tsing Hua University, Taiwan

  24. View Generation View 1 synthesis View 2 VCLAB, National Tsing Hua University, Taiwan

  25. View Generation View synthesis Disparity-based e.g. view1view3/2 view2 object(0,0) object(3,0) object(6,0) Depth-based 2D + depth map http://www.imec.be/ScientificReport/SR2007/html/1384302.html VCLAB, National Tsing Hua University, Taiwan 26 DEPTH MAPS EXTRACTION FROM MULTI-VIEW VIDEOS http://www.youtube.com/watch?v=KtRSbey1sKM

  26. View Generation VCLAB, National Tsing Hua University, Taiwan

  27. Display (3D space) Volumetric video Holographic video VCLAB, National Tsing Hua University, Taiwan http://www.youtube.com/watch?v=kIDgC2no1uo

  28. Display • Free viewpoint video • http://www.youtube.com/watch?v=vyhz8KgW49E • http://www.youtube.com/watch?v=GSumx0Zs2XA VCLAB, National Tsing Hua University, Taiwan

  29. Display with 3D Glasses • Stereo video with glasses • Left view and right view • Passive and active VCLAB, National Tsing Hua University, Taiwan

  30. Display with 3D Glasses Anaglyph, polarized, and shutter glasses http://www2.ciw.com.cn/h/2562/357866-17902.html VCLAB, National Tsing Hua University, Taiwan

  31. Display • Lenticular panel and Parallax barrier VCLAB, National Tsing Hua University, Taiwan

  32. Display • Stereo view display without glasses A2 A2 A2 Pixel 2 A2 A1 A2 A2 A1 A2 A1 A2 A1 A2 A1 A2 A1 Pixel 1 A1 A1 A1 A1 VCLAB, National Tsing Hua University, Taiwan

  33. Display • Multiview display without glasses C2 C2 C2 C1 Pixel 2 C2 C1 B2 C1 B2 B2 C1 B2 B1 A2 B1 A2 B1 A2 B1 Pixel 1 A2 A1 A1 A1 A1 VCLAB, National Tsing Hua University, Taiwan

  34. Conclusions • An overview of multiview video from capture to display. • Issues • Display • Reduction of brightness • Reduction of refresh rate • Reduction of temporal of spatial resolution • Coding efficiency • Decoding delay • 2D  3D • View synthesis • … VCLAB, National Tsing Hua University, Taiwan

  35. Reference A. Vetro, “Representation and Coding Formats for Stereo and Multiview Video,” Tech. Rep. TR2010-011, April, 2010. S. Gaël, “Depth Map Estimation and Use for 3DTV,” Tech. Rep. n0379, February, 2010. Y.-S. Ho and K.-J. Oh, “Overview of Multi-view Video Coding,” IWSSIP, 2007. 許精益 and 黃乙白 “3D立體顯示技術之發展與研究,” 光學工程第98期, 96年6月. VCLAB, National Tsing Hua University, Taiwan

More Related