1 / 41

Yong Seok Heo , Kyoung Mu Lee and Sang Uk Lee

Joint Depth Map and Color Consistency Estimation for Stereo Images with Different Illuminations and Cameras. Yong Seok Heo , Kyoung Mu Lee and Sang Uk Lee IEEE Transactions on Pattern Analysis and Machine Intelligence 2012. Overview. Introduction Relate Work Algorithm

asher
Télécharger la présentation

Yong Seok Heo , Kyoung Mu Lee and Sang Uk Lee

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Joint Depth Map and Color Consistency Estimation for Stereo Images with Different Illuminations and Cameras Yong SeokHeo,Kyoung Mu Leeand Sang UkLee IEEE Transactions on Pattern Analysis and Machine Intelligence 2012

  2. Overview • Introduction • Relate Work • Algorithm • Experimental Results

  3. Introduction (1/3) • image color values can be easily affected by radiometric variations including global intensity changesand local intensity changes (caused by varying light, vignettingand non-Lambertian surface) and noise. • For stereo matching, most algorithms assume radiometricallycalibrated images • However, there exist many real and practical situations or challenging applications in which radiometric variations between stereo images are inevitable.

  4. Introduction (2/3) • For examples: 3D reconstruction of aerial images [1], general multiview stereo [4], 3D modeling with internet photos (e.g. Photo Tourism [5] and Photosynth [6]), and PhotoModeler[7], etc.

  5. Introduction (3/3) • In general, color consistency and stereo matching are a chicken-and-egg problem. • Color consistency can enhance the performance of stereo matching, while accurate disparity maps can improve the color consistency or constancy. • In this paper, new iterative framework that infers both accurate disparity maps and colorconsistentimages for radiometrically varying stereo images are proposed.

  6. Relate Work • Stereo Matching: • Census transform (7 × 7) [41] • Mutual Information (MI) [22] • Adaptive Normalized Cross Correlation (ANCC) [15] • Color Consistency: • Color Histogram Equalized (CHE) [39]

  7. Non-parametric Local Transforms for Computing Visual Correspondence [41] R. Zabih and J. Woodfill, in Proc. European Conference on Computer Vision, 1994. • windows. • Rank Transform: • Non-parametric measure of local intensity. • Using the number of pixels in the local region whose intensity is less than the center pixel. • Census Transform: • Non-parametric summarizes of local image structure. • Using the number of neighboring pixels whose intensity is less than the center pixel.

  8. Stereo Processing by SemiglobalMatching and Mutual Information [22] H. Hirschmuller, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 328–341, 2008. : Joint probability distribution

  9. Robust Stereo Matching Using Adaptive Normalized Cross-Correlation [15] Y. S. Heo, K. M. Lee, and S. U. Lee, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 33, no. 4, pp. 807–822, 2011. • The whole window information around matching pixels is used by the NCC in order to find the mean and standard deviation. • Weight distribution information around matching pixels using the bilateral filter.

  10. Log-chromaticity Color Space [15](1/2) • : brightness factor • : illuminant color factor of channel • : gamma correction factor [15] Y. S. Heo, K. M. Lee, and S. U. Lee, “Robust stereo matching using adaptive normalized cross-correlation,” IEEE Trans. Pattern Analysis and Machine Intelligencea

  11. Log-chromaticity Color Space (2/2) • and are constant for each channel k • is an invariant color value for pixel p under radiometric variations. • M : multiplication factor which is set to 500.

  12. SIFT • The SIFT descriptor for each pixel is computed with this color value for each channel and with the intensity (gray) value of the original image . We denote and as SIFT descriptors.

  13. Joint Probability Density Function (1/2) • Joint probability density function (pdf) represents the statistical relationship between the left and right image color values. • compute the joint pdf by means of the SIFT descriptors rather than pixel values to encode the spatial gradient information.

  14. Joint Probability Density Function (2/2) • : normalization constant • SIFT-weighting factor : • :Euclidean distance • SIFT descriptors

  15. Linear Function Estimation • The log-chromaticity color space is linear, a linear function can be fitted to the joint pdf. • To find the linear functions, we use the Huber distance function ρ(r): • : the distance between the line and the Point • : the constant threshold parameter () • Using the OpenCVfunction ‘cvFitLine()’.

  16. Disparity Map Estimation (1/9) • : data energy • : smoothness energy • N : local four-neighborhood system • : the data cost that encodes the penalty for the dissimilarity of corresponding pixels • : the smoothness cost that penalizes the discontinuity of disparities between neighboring pixels. [34] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222–1239, 2001.

  17. Disparity Map Estimation (2/9) • We combine the mutual information and the SIFT descriptor in our data cost [11], [12]. • Where more weight is given to than to furtheremphasize features in the log-chromaticity color values. • The SIFT term has important role at the first iteration.

  18. Disparity Map Estimation (3/9) • and are mutual information (MI) termsfor log-chromaticity color and original gray images using

  19. Disparity Map Estimation (4/9) • In (10), is the adaptive weighting factor of a pixel p between and : [35] X. Hu and P. Mordohai, “Evaluation of stereo confidence indoors and outdoors,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2010.

  20. Disparity Map Estimation (5/9)

  21. Disparity Map Estimation (6/9) • MI tends to make some regions over-smooth, while SIFT can also blur some boundaries and is weak at textureless regions. • We incorporated segment-based plane-fitting constraints [36], [37] to produce sharper and more accurate disparity map. • In this framework, the disparity f is parameterized as . • Before extracting 3-D plane parameter for each segment, the mean-shift segmentation method [38] is applied to the left and right original color images, independently. [36] L. Hong and G. Chen, “Segment-based stereo matching using graph cuts,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2004. [37] J. Sun, Y. Li, S. B. Kang, and H.-Y. Shum, “Symmetric stereo matching for occlusion handling,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2005. [38] D. Comaniciu and P. Meer, “Mean shift: A robust approach toward feature space analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603–619, 2001.

  22. Disparity Map Estimation (7/9) • In the first step, initial plane-fitting is performed using only reliable pixels for each segment. • This solution is sensitive to outliers. Hence, using the estimated plane parameter, we re-compute disparity value in the range for each pixel:

  23. Disparity Map Estimation (8/9) • After the initial plane-fitting, we perform a refined plane-fitting scheme to find more accurate plane parameter for each segment s. • : the number of reliable pixels • : the number of pixels that have the same disparities as the disparity map obtained from (8).

  24. Disparity map estimation (9/9) • :weighting factor • : constant occlusion cost • Using (17) and (18), the total energy in (8) is minimized by the Graph-cuts expansion algorithm[34]. [34] Y. Boykov, O. Veksler, and R. Zabih, “Fast approximate energy minimization via graph cuts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222–1239, 2001.

  25. Fast approximate energyminimization via graph cuts [34] T. Gevers and H. Stokman, IEEE Trans. Pattern Analysis and Machine Intelligence, Jan. 2004. • The minimum cut problem is to find the cut with smallest cost. There are numerous algorithms for this problem with low-order polynomial complexity. • Using two move to refine that: swap and expansion.

  26. Occlusion Map Estimation [37] • We define which is a binary map for the left image. • is computed by warping to the left image, andassigning ‘0’ to visiting pixels, and ‘1’ to non-visitingpixels. • and : the weighting factors, set as 4.0 and 1.4. [37] J. Sun, Y. Li, S. B. Kang, and H.-Y. Shum, “Symmetric stereo matching for occlusion handling,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2005.

  27. Color Histogram Equalization (CHE) • The color histogram equalization (CHE) method was proposed in [39]. • : the total number of pixels in the image • = 255 : the maximum value of color • If is invariant under any illumination change, then the is also invariant. • This method is stable only under global radiometric changes such as exposure or gamma changes.

  28. Stereo Color Histogram Equalization (SCHE) • Where is the transformed value of using the estimated linear function from the joint pdf.

  29. Boosting the Disparity Map Estimation Using SCHE Images • The color consistency of the SCHE images is based on the accurate disparity map estimation. • Conversely, the estimation of the disparity map can benefit from the color-consistent SCHE stereo images.

  30. Experimental Results • Using various test images with ground truth disparity map such as Aloe, Dolls, Moebius, Art, Laundry, Reindeer, Rocks1, and Cloth4 dataset in [40] that have different radiometric variations. • Each data set has three different camera exposures (0~2) and three different configurations of the light source (1~3). • The total running time of our method for Aloe images (size : 427 × 370, disparity range : 0-70), for example, is about 4 minutes on a PC with Intel(R) Core(TM) i7-2600K 4.5GHz CPU. “http://vision.middlebury.edu/stereo/,” 2012.

  31. Experimental for SCHE

  32. Color consistency performance • Computed the RMSE [39] images performed individually for the left and right original images. • ‘CHE1’ means RMSE for the stereo images after individually performing CHE using original input stereo images. • ‘CHE2’ means RMSE for the stereo images after individually performing CHE using stereo images in the log-chromaticity color space. [39] G. Finlayson, S. Hordley, G. Schaefer, and G. Y. Tian, “Illuminant and device invariant colour using histogram equalisation,” Pattern Recognition, vol. 38, no. 2, pp. 179–190, 2005.

  33. Effects of SIFT Weight in MI Computation • We turned off both the SIFT term in (10) and the plane-fitting constraint in (17) from our data cost.

  34. Effects of Adaptive Weight • Compared the results using this preliminary data cost in (10) by varying which is fixed as the same value for all pixels.

  35. Stereo Matching Performance Comparison • Census transform [41] • Mutual Information (MI) [22] • Normalized Cross Correlation (NCC) and Adaptive Normalized Cross Correlation (ANCC) [15]. • To evaluate the effects of exposure changes, we only changed the index of exposure.

  36. Stereo Matching Performance Comparison

  37. Different Configurations of the Light Source • Only changed the index of the light configuration while fixing the exposure.

  38. Tests for Scenes With Different Cameras • The left images were taken by Canon IXUS 870 IS, and the right images were taken by Sony Cyber-shot DSC-W570. • The left images were taken with flash, while the right images were taken without flash. • The exposure times of the left and right images were set 1/60 sec and 1/100 sec, respectively.

More Related