Create Presentation
Download Presentation

Download

Download Presentation

Qingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE,

179 Views
Download Presentation

Download Presentation
## Qingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE,

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Stereo Matching with Color-Weighted Correlation,**Hierarchical Belief Propagation,and Occlusion Handling Qingxiong Yang, Student Member, IEEE, Liang Wang, Student Member, IEEE, Ruigang Yang, Member, IEEE, HenrikStewe´ nius, Member, IEEE, and David Niste´ r, Member, IEEE IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 31, NO. 3, MARCH 2009**Outline**• Introduction • System Overview • Methods • Initialization • Pixel Classification • Iterative Refinement • Fast-Converging Belief Propagation • Depth Enhancement • Experiments • Conclusion**Introduction**• Stereois one of the most extensively researched topics in computer vision. • Energy Minimization framework: • Graph Cut • Belief Propagation(BP)**Objective(Contribution)**• To formulate stereo model with careful handling of: • Disparity • Discontinuity • Occlusion • Differs from the normal framework in the final stages of the algorithm • Outperforms all other algorithms on the average**Initialization**• Input: • Left Image IL • Right Image IR • Output: • Initial Left Disparity Map DL(0) • Initial Right Disparity Map DR • Initial Data Term ED(0) • CL • CR • ED(0) • DR • DL(0)**Initialization**• Color-weighted Correlation • To build the Correlation Volume • Makes the match scores less sensitive to occlusion boundaries • By using the fact that occlusion boundaries most often cause color discontinuities as well • CL • CR • ED(0) • DR • DL(0)**Correlation Volume**• Color difference Δxybetween pixel x and y (in the same image) Ic: Intensity of the color channel c • The weight of pixel x in the support window of y: • 10 • 21 • Color Difference • Spatial Difference**Correlation Volume**• The Correlation Volume[27]: • Wx :support window around x • d(yL,yR) : pixel dissimilarity[1] • xL , yL : pixels in left image IL • xR, yR: corresponding pixels in right image IR • dx : disparity value of pixel XL in IL • Dissimilarity[1] • weight • Pixels in the window • weight xR = xL – dx yR= yL– dx dx = arg min CL,x(yL,yR)**Correlation Volume**• Bad Pixel • Disparity Map [1] S. Birchfield and C. Tomasi, “A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, pp. 401-406, 1998. [27] K.-J. Yoon and I.-S. Kweon, “Adaptive Support-Weight Approach for Correspondence Search,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, pp. 650-656, 2006.**Initialization**• Initial Data Term • Total energy = Data Term + Smooth Term • Computed from Correlation Volume • Given an iteration index i= 0here because it will be iteratively refined • CL • CR • ED(0) • DR • DL(0)**Initial Data Term**• Initial Data Term: • Ƞbp : twice the average of correlation volume to exclude the outliers • Average Correlation Volume • Correlation Volume • X 2 • 0.2**Initialization**• hierarchical Belief Propagation • Employed with the data term and the reference image • Resulting in the initial left and right disparity mapsDL(0) and DR • CL • CR • ED(0) • DR • DL(0)**Pixel Classification**• Output • Input**Pixel Classification**• Mutual Consistency Check • Requires that the disparity value from the left and right disparity maps are consistent, i.e., • Not Pass : occluded pixel • Pass : unoccluded pixel =>Correlation Confidence Measure**Pixel Classification**• Correlation Confidence • Based on how distinctive the highest peak in a pixel's correlation profile is • : the cost for the best disparity value • : the cost for the second best disparity value • 0.04 • If > αs stable • Else unstable dx = arg min CL,x(yL,yR)**Input**• Goal: to propagate information from the stable pixels to the unstable and the occluded pixels • Iteration**Iterative Refinement**• Color Segmentation • Color segments in IL are extrated by Mean Shift[6] [6] D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach Toward Feature Space Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, pp. 603-619, 2002.**Iterative Refinement**• Plane Fitting • Using the disparity values for the stable pixels in each color segment • Disparity values are taken from the current hypothesisfor the left disparity mapDL(i). (Initial:DL(0)) • The plane-fitted depth map is used as a regularization for the new disparity estimation.**Iterative Refinement**[10] M.A. Fischler and R.C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Comm. ACM, vol. 24, pp. 381-395, 1981. • Plane Fitting • Using RANSAC[10] • Iterates until the plane parameters converge**Iterative Refinement**• Plane Fitting output : D(i) • The ratio of stable pixels of each segment: • If Ratio > ȠS • Stable pixels: D(i) • Unstable, Occluded pixels: D(i) • If Ratio ≤ȠS • All pixels : D(i) pf • 0.7 L pf pf**Iterative Refinement**• Absolute Difference: • D(i+1): New Disparity Map • D(i): Plane-fitted Disparity Map • Data Term: L pf • 2.0 • 0.5 • 0.05**Belief Propagation**• The core energy minimization of our algorithm is carried out via the hierarchical BP algorithm. • Total Energy for Pixel X • Data Term • Smooth Term**Max-Product Belief Propagation**• Max-Product BP[25] : • : Message vector passed from pixel X to one of its neighbors Y • Data Term • Jump Cost [25] Y. Weiss and W. Freeman, “On the Optimality of Solutions of the Max-Product Belief PropagationAlgorithm in Arbitrary Graphs,” IEEE Trans. Information Theory, vol. 2, pp. 732-735, 2001.**Max-Product Belief Propagation**• Jump Cost: • dx:Disparity of pixel X • d:Disparity of pixel Y (X’s neighbor) • αbp: The number of disparity levels / 8 • ρs: 1 – (normalized average color difference) • ρbp: The rate of increase in the cost • Disparity Difference • of pixel X and its neigbor Y • 1**Max-Product Belief Propagation**• Total Energy for pixel X: • Finally the label d that minimizes the total Energy for each pixel is selected. • Data Tem • Smooth Tem**Hierarchical Belief Propagation**• Standard loopy BP algorithm is too slow. • Hierarchical BP[9] runs much faster while maintaining comparable accuracy. • Works in a coarse-to-fine manner [9] P.F. Felzenszwalb and D.P. Huttenlocher, “Efficient Belief Propagation for Early Vision,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 261-268, 2004.**Hierarchical Belief Propagation**• Finer • (Level 0) • Coarser • (Level 1)**Fast-Converging Belief Propagation**• Alarge number of iterations is required to guarantee convergence in a standard BP algorithm. • Fast-Converging BP effectively removes the redundant computation. • Only updating the pixels that have not yet converged (value bigger than ȠZ ) • 0.1**Depth Enhancement**• To reduce the discontinuities caused by the quantization • Sub-pixel Estimation algorithm is proposed. • Cost Function:**Depth Enhancement**• The depth with the minimum of the cost function: • d: the discrete depth with the minimal cost • d+: d+1 • d- : d-1 • Replace each value with the average of those values that are within one disparity over a 9 x 9 window**Experiments**Parameter Settings Used Throughout:**Experiments**Parameter Settings Used Throughout:**Experiments**Results on the Middlebury Data Sets with Error Threshold 1 Error% nonocc : The subset of the nonoccludedpixels disc :The subset of the pixels near the occluded areas. all : The subset of the pixels being either nonoccludedor half-occluded