Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University
Processing Mosaics
Redundant Measurements Generalized Mosaicing: Yoav Schechner and Shree Nayar
Mantis Shrimp C Takata Scanning with Less Redundancy Different rows = Different optical characteristics
Generalized Mosaicing Brightness Dynamic Range Spectrum Field of View Depth of Field Polarization camera Spatially varying filter Schechner, Nayar, Generalized mosaics
88 - 18,794 Mosaic + High Dynamic Range
Fusion of Measurements Maximum-Likelihood Solution Schechner, Nayar, Generalized mosaics
-2 -6 10 10 -2 1 -4 log M 2 -6 8 bits (almost) -8 x (pixels) 8 bit camera Dynamic range as 16 bits = 1 Log of the mask Generalized Mosaicing: Yoav Schechner and Shree Nayar
M Consistency, Smoothness A average row consistency constraints x Mask Self-Calibration Unknown filter (vignetting) Schechner, Nayar, Generalized mosaics
Image Registration Schechner, Nayar, Generalized mosaics
Image Registration Bias towards “no motion” Schechner, Nayar, Generalized mosaics
Uncertainty Image Registration Schechner, Nayar, Generalized mosaics
motion of 2 pixels 5 pixels 4 pixels 10 pix 9 pix 18 pix 19 pix However, we need: multiscale UNCERTAINTIES Registration: Standard “Coarse to Fine” Schechner, Nayar, Generalized mosaics
estimate -1 2 Maximum-Likelihood Pyramid At each levelL , for each pixel Schechner, Nayar, Generalized mosaics
ML estimation Max-Likelihood Pyramid Gaussian Pyramid
Generalized Mosaic log I log I frame Intensity range Intensity range frame x Spatial range Spatial range Mosaic x Schechner, Nayar, Generalized mosaics
spectral y l x Variable Spectral Filter Generalized Mosaicing: Yoav Schechner and Shree Nayar
MultiSpectral Mosaic 400 400 500 500 600 600 700 700 400 500 600 700 l l l Generalized Mosaicing: Yoav Schechner and Shree Nayar
sunset fluorescent Rendering: Any Illumination Generalized Mosaicing: Yoav Schechner and Shree Nayar
Illumination at a Glance Halogen Fluorescent y y l l x x Extra information • Illuminant spectrum
Spatially Varying Polarizer Polarization Mosaicing: Yoav Schechner and Shree Nayar
polarizance 1 90 o o polarizing angle 60 0.8 o 30 0.6 o 0 0.4 o -30 transmittance 0.2 o -60 o -90 0 Spatially Varying Polarizer Polarization Mosaicing: Yoav Schechner and Shree Nayar
Raw images Polarization Mosaicing: Yoav Schechner and Shree Nayar
reflected structure Schechner Shamir Kiryati JOSA-A 2000 transmitted painting Polarization Mosaic
Insert aprismbehind the lens y = Depth x Spatially Varying Focus Generalized Mosaicing: Yoav Schechner and Shree Nayar
All Focused Mosaic … and a Depth Map Generalized Mosaicing: Yoav Schechner and Shree Nayar
Multi-dimensional Mosaics - Simultaneously What else? Dynamic Range & Spectrum & Polarization etc. Generalized Mosaicing: Yoav Schechner and Shree Nayar
l • Signal undergoing LPF • “Band limited” signal • Minimizing aliasing • Nyquist sampling rate # samples M 11111111 1 1 1/2 2 1/4 3 9 11111111? length L 1 M 10111001?? bandwidth B 1011100??? resolution Dl 101110???? aperture D Sampling Criteria Number of samples per scene point 1011100101 Generalized Mosaicing: Yoav Schechner and Shree Nayar
Mosaic Generalized Mosaic Dynamic range Spatially varying filter I Spectrum l 88 – 18,794 Polarization Depth + Focus Wide field of view
