1 / 43

Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction

This research focuses on developing a black box solver that can jointly estimate object class segmentation and dense stereo reconstruction. The approach involves assigning disparity labels to each pixel based on a disparity set and using unary and pairwise potentials to encourage label consistency. The method utilizes graph-cut based range-move inference to achieve accurate results.

baxtere
Télécharger la présentation

Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Joint Optimisation for Object ClassSegmentation and Dense StereoReconstruction Ľubor Ladický, Paul Sturgess, Christopher Russell, Sunando Sengupta, Yalin Bastanlar, William Clocksin, Philip H.S. Torr Oxford Brookes University http://cms.brookes.ac.uk/research/visiongroup/

  2. Joint Object Class Segmentationand Dense Stereo Reconstruction Black Box Solver Left Camera Image Object Class Segmentation Right Camera Image Dense Stereo Reconstruction

  3. Joint Object Class Segmentationand Dense Stereo Reconstruction Objective : Joint Estimation Black Box Solver Left Camera Image Object Class Segmentation Right Camera Image Dense Stereo Reconstruction

  4. Dense Stereo Reconstruction • For each pixel assigns a disparity label : y • Disparities from the discrete set {0, 1, .. D} Left Camera Image Right Camera Image Dense Stereo Result

  5. Dense Stereo Reconstruction Unary Potential Disparity = 0 Unary Cost dependent on the similarity of patches, e.g.cross correlation

  6. Dense Stereo Reconstruction Unary Potential Disparity = 5 Unary Cost dependent on the similarity of patches, e.g.cross correlation

  7. Dense Stereo Reconstruction Unary Potential Disparity = 10 Unary Cost dependent on the similarity of patches, e.g.cross correlation

  8. Dense Stereo Reconstruction Unary Potential Disparity = 15 Unary Cost dependent on the similarity of patches, e.g.cross correlation

  9. Dense Stereo Reconstruction Pairwise Potential • Encourages label consistency in adjacent pixels • Cost based on the distance of labels Linear Truncated Quadratic Truncated

  10. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution

  11. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution After 1st expansion Final solution

  12. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution After 1st expansion After 2nd expansion

  13. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution After 1st expansion After 2nd expansion After 3rd expansion

  14. Dense Stereo Reconstruction • Graph-Cut based Range-move inference (Kumar et al. NIPS09, Veksler et al. CVPR09) Original Image Initial Solution After 1st expansion After 2nd expansion After 3rd expansion Final solution

  15. Dense Stereo Reconstruction Does not work for Road Scenes ! Dense Stereo Reconstruction Original Image

  16. Dense Stereo Reconstruction Does not work for Road Scenes ! Different brightness in cameras Patches can be matched to any other patch for flat surfices

  17. Dense Stereo Reconstruction Does not work for Road Scenes ! Different brightness in cameras Patches can be matched to any other patch for flat surfices Could object recognition for road scenes help? Recognition of road scenes is relatively easy (Sturgess et al., BMVC09)

  18. Object Class Segmentation • Aims to assign a class label for each pixel of an image • Classifier trained on the training set • Evaluated on never seen test images

  19. Object Class Segmentation Unary Potential • Likelihood of a pixel taking a label (Shotton et al. ECCV06, He et al, CVPR04, Ladický et al. ICCV 09)

  20. Object Class Segmentation Pairwise Potential • Contrast sensitive Potts model • Encourages label consistency in adjacent pixels

  21. Object Class Segmentation Higher Order Potential • Encouraging consistency in superpixels (Kohli et al. CVPR08) • Merging information at different scales (Ladický et al. ICCV09)

  22. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass Original Image Initial solution

  23. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass building grass Original Image Initial solution Building expansion

  24. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass building grass Original Image Initial solution Building expansion sky building grass Sky expansion

  25. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass building grass Original Image Initial solution Building expansion sky sky tree building building grass grass Sky expansion Tree expansion

  26. Object Class Segmentation • Graph-Cut based α-Expansion inference (Boykov et al. ICCV99) grass building grass Original Image Initial solution Building expansion sky sky sky tree tree building building building aeroplane grass grass grass Sky expansion Tree expansion Final Solution

  27. Object Class Segmentation vs.Dense Stereo Reconstruction  • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) sky

  28. Object Class Segmentation vs.Dense Stereo Reconstruction • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buildings & pedestrians have their typical height sky building car

  29. Object Class Segmentation vs.Dense Stereo Reconstruction • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane sky building car road

  30. Object Class Segmentation vs.Dense Stereo Reconstruction • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane • Buildings and pavement on the sides sky building car road

  31. Object Class Segmentation vs.Dense Stereo Reconstruction • Object class and 3D location are mutually informative • Sky always in infinity (disparity = 0) • Cars, buses & pedestrians have their typical height • Road and pavement on the ground plane • Buildings and pavement on the sides • Both problems formulated as CRF • Joint approach possible? sky building car road

  32. Joint Formulation • Each pixels takes label zi = [ xi yi ] L1  L2 • Dependency of xi and yi encoded as a unary and pairwise potential, e.g. • strong correlation between x = road, y = near ground plane • strong correlation between x= sky, y = 0 • Correlation of edge in object class and disparity domain

  33. Joint formulation Unary Potential Object layer Joint unary links Disparity layer • Weighted sum of object class, depth and joint potential • Joint unary potential based on histograms of height

  34. Joint Formulation Pairwise Potential Object layer Joint pairwise links Disparity layer • Object class and depth edges correlated • Transitions in depth occur often at the object boundaries

  35. Joint Formulation

  36. Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [xL1, yL2], Possible in case of metric pairwise potentials Inference

  37. Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [xL1, yL2], Possible in case of metric pairwise potentials Inference Too many moves! ( |L1| |L2| ) Impractical !

  38. Projected move for product label space One / Some of the label components remain(s) constant after the move Set of projected moves α-expansion in the object class projection Range-expansion in the depth projection Inference

  39. Leuven Road Scene dataset Contained 3 sequences 643 pairs of images We labelled 50 training + 20 test images Object class (7 labels) Disparity (100 labels) Available on our website http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip Dataset Left camera Right camera Object GT Disparity GT

  40. Qualitative results Original Image Object GT Object Result Disparity GT Disparity Alone Disparity Jointly • Large improvement for dense stereo estimation • Minor improvement in object class segmentation

  41. Quantitative disparity results Dependency of the ratio of correctly labelled pixels within the maximum allowed error delta

  42. Application to monocular sequences Making method (close to) real time Application to multi-view problems Optical flow / motion estimation On-going and Future Work

  43. First dataset with both object class and disparity labels Joint estimation improves significantly disparity results Projected moves make inference much faster Questions ? Summary

More Related