Extracting Depth and Matte using a Color- Filtered Aperture - PowerPoint PPT Presentation

extracting depth and matte using a color filtered aperture n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Extracting Depth and Matte using a Color- Filtered Aperture PowerPoint Presentation
Download Presentation
Extracting Depth and Matte using a Color- Filtered Aperture

play fullscreen
1 / 97
Extracting Depth and Matte using a Color- Filtered Aperture
156 Views
Download Presentation
molimo
Download Presentation

Extracting Depth and Matte using a Color- Filtered Aperture

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Extracting Depth and Matteusing a Color-Filtered Aperture Yosuke Bando TOSHIBA + The University of Tokyo Bing-Yu Chen National Taiwan University Tomoyuki Nishita The University of Tokyo

  2. Outline • Background • Related Work • Our Method • Results • Conclusion

  3. Computational Cameras • Capture various scene properties • High dynamic range, high resolution, • Large field of view, reflectance, depth, • ... and more • With elaborate imaging devices • Camera arrays • Additional optical elements [Wilburn 2005] [Nayar 1997]

  4. Compact Computational Cameras • Small devices • Simple optical elements [Ng 2005] [Levin 2007]

  5. Our Goal • Capture scene properties • With minimal modification to the camera

  6. Our Goal • Capture scene properties • Depth maps • Alpha mattes • With minimal modification to the camera Captured image Depth Matte

  7. Our Goal • Capture scene properties • Depth maps • Alpha mattes • With minimal modification to the camera • Put color filters in a camera lens aperture Captured image Depth Matte Lens with color filters

  8. Our Goal • Capture scene properties • Depth maps • Alpha mattes • With minimal modification to the camera • Put color filters in a camera lens aperture • This idea itself is not new Captured image Depth Matte Lens with color filters

  9. Contents • Background • Related Work • Our Method • Results • Conclusion

  10. Previous Color-Filter Methods • Extract (only) depth maps • With low precision • Or, a specialized flashbulb is used • Spoils the visual quality of captured images [Amari 1992] [Chang 2002]

  11. Coded Aperture • Patterned mask in the aperture • Changes frequency characteristics of defocus • Facilitates blur identification/removal [Levin 2007] [Veeraraghavan 2007] Captured image Amount of defocus blur ( depth) Lens with a mask

  12. Single-Lens Multi-View Capture • Records light rays separately depending on their incident angle • Enables light field rendering [Ng 2005] [Veeraraghavan 2007] [Adelson 1992] [Georgeiv 2006] [Liang 2008]

  13. Matting • Automatic matting by multiple cameras [McGuire 2005] Video matting 3 cameras with half mirrors [Joshi 2006] Array of 8 cameras Video matting

  14. Our Method • Features • Automatic depth and matte extraction • Single hand-held camera • Single shot • Contributions 1. Improved depth estimation 2. Novel matting algorithm for images captured thru a color-filtered aperture

  15. Outline • Background • Related Work • Our Method • Results • Conclusion

  16. Our Method • Color-filtered aperture • Depth estimation • Matting

  17. Our Method • Color-filtered aperture • Depth estimation • Matting

  18. Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens

  19. Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Aperture part of the disassembled lens

  20. Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45

  21. Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 Our prototype lens with color-filters

  22. Our Prototype Camera Lens • Took me just a few hours to fabricate • Using a micro-screwdriver and a box cutter Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 Our prototype lens with color-filters

  23. Captured Image

  24. Red Plane

  25. Green Plane

  26. Blue Plane

  27. Captured Image • Has depth-dependent color-misalignment • NOT due to chromatic aberration

  28. Why Do Colors Get Misaligned? Color filters Background Foreground object Image sensor Lens

  29. Why Do Colors Get Misaligned? Color filters Background Foreground object Image sensor Lens

  30. Why Do Colors Get Misaligned? Color filters Background Foreground object Image sensor Lens Color filters Background Image sensor Lens

  31. Why Do Colors Get Misaligned? Color filters Background Foreground object Image sensor Lens Color filters Background Image sensor Lens

  32. Our Method • Color-filtered aperture • Depth estimation • Matting

  33. Depth Estimation • Our camera captures 3 views in the RGB planes •  Stereo reconstruction problem Red plane Green plane Blue plane

  34. Depth Estimation • Our camera captures 3 views in the RGB planes •  Stereo reconstruction problem • However, their intensities don’t match • Contribution 1: improved correspondence measure between the RGB planes Red plane Green plane Blue plane

  35. Original Image

  36. Disparity = 1

  37. Disparity = 2

  38. Disparity = 3

  39. Disparity = 4

  40. Disparity = 5

  41. Disparity = 6

  42. When Is The Color Aligned? Disparity

  43. Color-Alignment Measure • Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006]

  44. Color-Alignment Measure • Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006]

  45. Color-Alignment Measure • Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006]

  46. Color-Alignment Measure • Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006]

  47. Color-Alignment Measure • Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 1 pixel Disparity = 0 Disparity = 1

  48. Color-Alignment Measure • Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 1 pixel Disparity = 0 Disparity = 1

  49. Color-Alignment Measure • Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 3 pixels Disparity = 0 Disparity = 1 Disparity = 3

  50. Color-Alignment Measure • Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 3 pixels Disparity = 0 Disparity = 1 Disparity = 3