The plan for today

1. The plan for today • Camera matrix • Part A) Notation, preprocessing, and basic concepts. • Part B) 4 Stereo Algorithms Slides are courtesy of Prof. Ronen Basri

2. Stereo Vision • Objective: 3D reconstruction • Input: 2 (or more) images taken with calibrated cameras • Output: 3D structure of scene • Steps: • Rectification • Matching • Depth estimation

3. Rectification • Image Reprojection • reproject image planes onto common plane parallel to baseline • Notice, only focal point of camera really matters (Seitz)

4. Rectification • Any stereo pair can be rectified by rotating and scaling the two image planes (=homography) • We will assume images have been rectified so • Image planes of cameras are parallel. • Focal points are at same height. • Focal lengths same. • Then, epipolar lines fall along the horizontal scan lines of the images

5. Cyclopean Coordinates • Origin at midpoint between camera centers • Axes parallel to those of the two (rectified) cameras

6. Disparity • The difference is called “disparity” • dis inversely related to Z: greater sensitivity to nearby points • dis directly related to b: sensitivity to small baseline

7. Main Step: Correspondence Search • What to match? • Objects? More identifiable, but difficult to compute • Pixels? Easier to handle, but maybe ambiguous • Edges? • Collections of pixels (regions)?

8. Random Dot Stereogram • Using random dot pairs Julesz showed that recognition is not needed for stereo

9. Random Dot in Motion

10. Finding Matches

11. 1D Search • More efficient • Fewer false matches SSD error disparity

12. Finding Matches • Under what conditions pixels can be matched? • Ignoring specularities, we can assume that matching pixels have the same brightness (constant brightness assumption) • Still, changes in gain and sensitivity may change the values of pixels

13. Possible solutions • Use larger windows • Constraint the search • Other metric e.g., Normalized correlation

14. Window Size W = 3 W = 20

15. Constraining the Search • Restrict search to epipolar lines (1D search) • Enforce ordering Problem: not always true • Enforce smoothness Problem: discontinuities at object boundaries

16. Ordering

17. Ordering

18. Summary basic ideas • Restrict search to epipolar lines (1D search) • Use larger elements (larger windows, edges, regions) Problem: large elements may be distorted • Enforce ordering Problem: not always true • Other similarity measures (e.g., Normalized correlation) • Enforce smoothness Problem: discontinuities at object boundaries

19. Comparison of Stereo Algorithms D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," International Journal of Computer Vision,47 (2002), pp. 7-42. Scene Ground truth

20. Scharstein and Szeliski

21. Results with window correlation Window-based matching (best window size) Ground truth

22. Graph Cuts Graph cuts Ground truth

23. Correspondence as Optimization • Most stereo algorithms attempt to minimize a functional that usually consists of two terms: where - penalizes for quality of a match (unary) - penalizes non smooth (or even non fronto-parallel) reconstructions (binary) • Many different optimization approaches were proposed

24. Part B) 4 Stereo Algorithms • Dynamic programming • Minimal cut/Max flow • Space carving • Graph cut optimization

25. ?

26. 1D Methods: Dynamic Programming • Discretize the 3-D space • Find the correct curve at every slice (A slice = epipolar plane)

27. Dynamic programming Find correspondences of each epipolar line separately

28. Dynamic programming

29. Dynamic programming How do we find the best curve? • Assign weight of all edges insertion match deletion

30. Dynamic programming How do we find the best curve? • Assign weight of all edges • Find shortest path • Dijkstra insertion match deletion

31. Results

32. Dynamic programming Advantages • Simple, efficient • Globally optimal Disadvantages • Each slice computed independently (smoothness is not enforced between slices) • Problems due to discretization (tilted planes)

33. Stereo Algorithms Brief review of 4 algorithms: • Dynamic programming • Minimal cut/Max flow • Space carving • Graph cut optimization

34. Min Cut/Max Flow

35. Min Cut/Max Flow

36. Min Cut/Max Flow

37. Min Cut/Max Flow

38. Min Cut/Max FlowMain idea: Lets solve all DP problem together. Objective: find the optimal cut using all the slices simultaneously.

39. Min Cut/Max Flow

40. Min Cut/Max Flow Construct a graph: • Every voxel (3-D point in space) is a node • Every node is connected to its 6 neighbors

41. Min Cut/Max Flow Weights on the edges: • Data cost: change in pixel value Neighbor Innext slice/row data Neighbor Neighbor data Neighbor Innext slice/row

42. Min Cut/Max Flow Weights on the edges: • Data cost: change in pixel value • Smoothness cost: change in depth smooth smooth smooth smooth

43. Min Cut/Max Flow Weights on the edges: • Data cost: change in pixel value • Smoothness cost: change in depth data

44. … … Min Cut/Max Flow Source • Add source and sink • Find min cut ∞ ∞ Sink

45. Min Cut/Max Flow Data penalty Smoothness penalty

46. Results Input Min cut Dynamic programming

47. Min Cut/Max Flow Advantages • All slices are optimized simultaneously • Efficient Disadvantages • Extension to multi-camera is difficult • Discretization

48. Space Carving • Multi-view stereo • Every point in space corresponds to a match in the images • Compute data term for each match

49. Space Carving • Multi-view stereo • Every point in space corresponds to a match in the images • Compute data term for each match (“photo-consistency”)

50. Space Carving • Dynamic data term (taking occlusion into account) • Order of sweep is important